Darktable Camera Color Profiling
In the past I wrote about profiling for camera profiling for UFRaw. Since then I’ve pretty much switched to Darktable, and I’ve learned a few things. So here we go again, with a vengeance…
Why?
The pretty images our camera’s output aren’t literally what the camera sensors see, there is a lot of proprietary postprocessing involved. Now when working in the RAW format we ditch all that postprocessing in favor of RAW sensor data. Which is good if we want maximum control and flexibility. The problem is that camera vendors don’t publish their proprietary postprocessing methods, which leaves us with a problem, how do we postprocess then?
In the past some good work on this topic has been done by Adobe, who published their DNG specification including color matrices. Color matrices are specifications on how camera native color is transformed into something that an end user might like, and ideally will be correct when viewed on a calibrated display.
Historically there have been some problems with the Adobe color matrices, the biggest of these is the rendition of red colors. To this day I still have no clue why this really is a problem. The only explanation I can think of is that this has been a deliberate compromise on Adobe’s part to safely render skin tones, at the cost of red rendition elsewhere. This is speculation at best…
So ideally we want an alternate enhanced color matrix that will render red as it’s supposed to, possibly at the cost of some skin tones rendition (rendition of skin tones is often a matter of taste, so we can fix that using the color zones “natural skin tones” preset).
Types of profiles
Without going into all the nitty gritty details, there are basically two kinds of profiles, the first are XYZ matrices (these are often called color matrices) and are typically combined with a gamma curve, the second are LUT profiles. What’s a LUT, it’s a lookup table… The big difference is that with a XYZ matrix all color transformations are calculated on the fly, while LUTs are precalculated, so transforming color via a LUT is simply looking up an input color and it’s matching output color. The nice thing about LUTs is that they can deal with slight (nonlinear) deviations and can even be tweaked for creative purposes. So when generating a LUT profile, the profile is likely to pick up some of the peculiarities of your particular camera. XYZ matrices don’t have that problem since they are defined by only 3×3 coordinates in XYZ colorspace, and thus are quite generic by their very nature. This is also the reason why we are sticking to color matrices instead of supplying more detailed LUT profiles (besides diskspace usage).
What digital camera’s can be color profiled
All camera’s that can output a digital RAW format supported by RawSpeed can be properly profiled. This effectively means that most compact camera’s can not be profiled (yes, some could using the CHDK firmware hack). Probably half of all digital bridge camera’s can output RAW and thus can be profiled as well. Pretty much all digital SLR camera’s are covered as well…
What color target do I need?
There are currently lots of color targets available on the market produced using different techniques and sold at very different prices. Here are some of the things you need to keep in mind…
First we can classify all targets into two groups, matte and semigloss… In theory semigloss targets can cover a larger gamut (range of colors) than matte targets, but most semigloss targets tend to glare, which is a big inconvenience when shooting the target. Most IT8 targets are semigloss, CMP’s Digital Target for example is matte… I personally prefer matte targets for their convenience since they are easily shot using a decent strobe, which is often hard to do with semigloss target. Semigloss targets are often shot best outdoors on a sunny non-cloudy day at an angle to prevent glare.
Second there’s the patch count, Gretag’s Classic ColorChecker only has 24 patches, while most IT8 targets have up to 288 patches, and CMP’s Digital Target even has 570 patches. When one only want to generate a color matrix this is not of particularly great relevance, the ColorChecker’s 24 patches are enough. But if one wants to do LUT profiles more patches tends to be better.
Third there’s the production technique, some charts like CMP’s Digital Target are produced on highend inkjets, which implies the target is very vulnerable to moisture making it more fragile than other targets. Most IT8 targets seems to be produced using traditional minilabs, and last but not least some (typically Gretag’s) targets are painted. In theory painted target should be superior in the sense that the manufacturer has greatest spectral control over the patches, which means you’re less vulnerable to metamerism when shooting the target under different lighting conditions.
Fourth there’s reference measurements, some targets like Gretag’s ColorChecker are very accurately produced to a single reference specification. Most IT8 targets are produced in batches, and will include per batch average reference data. The last and most accurate approach are individually measured charts, which means each chart has a reference which is specific to that chart alone, CMP does this for all it’s Digital Target’s and some IT8 vendors will also offer such services on request.
And last and least, don’t try to print your own target, even if you have a color calibrated printer, it will not be accurate enough!
Buying Advice
If you want a target that you can easily use with commercial software as well you should probably buy a (relatively expensive) ColorChecker Passport.
If you want a convenient and highly accurate target you should probably buy an (expensive) CMP Digital Target. This is my current personal preference.
If you want an affordable high quality target, and don’t mind some extra fiddling to get the shot right, you could probably buy an IT8 target from Wolf Faust.
Small note, Wolf Faust really kicks ass, since he relicensed his reference files under the terms of the Creative Commons Attribution-Share Alike license.
Shooting the color target
If you already have a target, or just purchased or borrowed one, we can try to shoot the target. First check if the target is clean and has no damage to the color patches. Then depending on the type of target you have available we can determine the shooting conditions:
Shooting a matte target
If you’re lucky enough to have a matte target available shooting it will be a breeze, since you can shoot the target head-on using a decent strobe. When possible put a dark grey or black piece of card behind the target. When I said decent strobe I meant something like a proper branded strobe like Canon, Nikon, Metz, etc. Please don’t do this with your generic asian no-brand flashes, since we don’t really know the quality of the light it produces, it might be fine for your photos, but I’d rather not use it for profiling purposes…
Shooting a semigloss target
If you have semigloss target the current recommended way to shoot it, is to wait for a sunny day with little or no clouds. Then usually at noon when the sun is highest in the sky shoot the target when there are no clouds obscuring the sun. When possible put a dark grey or black piece of card or cloth behind the target. You probably need to position the target at an angle (with regard to the camera’s position) to prevent any glare…
… Shoot the target using a modern normal lens (no antiques (since lens coating can differ) or fisheye folks) at about 50mm or so, at a distance of about roughly 1 meter or 3 feet. The target should cover about 50-75% of the image surface. The dark piece of board behind the target should be visible on all four sides of the target.
Now comes the hard part, getting the exposure right… Unless your camera has a true RAW histogram, the histogram will be useless to you. The best exposure will likely look slightly overexposed as a preview image. The best way to approach this is to take a boatload of exposures from severely underexposed to severely overexposed and select the best exposure later on using Darktable.
So, if applicable set your strobe to full manual (1/2 strength for GN36, 1/4 strength for GN48, 1/8 strength for higher GNs). Then set your camera to full manual mode as well, set Exposure to 1/200 sec, and set the sensor sensitivity to it’s normal lowest possible setting (without pushing), which is usually ISO 100 or ISO 200. Don’t use any special modes which allow extra low or extra high ISO, since this is often digital trickery… Then close down the aperture all the way (usually f32 or something) and take a shot, open up the aperture 1/3rd stop and take another stop, keep opening up the aperture and taking shots all the way, until the aperture is fully open. Usually this means you have taken about 20-30 shots by now.
Selecting and Processing a target image in Darktable
Now import the images you shot into Darktable, pick a target image which looks slightly overexposed, and open it in darkroom (develop) mode. Then disable all plugins except for these:
- raw import
- exposure
- white balance
- highlight reconstruction
- input color profile (set to linear rgb (or rec709 in newer versions)/absolute colorimetric)
- output color profile (set both to linear rgb (rec709 in newer versions)/absolute colorimetric)
- display color profile (set both to linear rgb (rec709 in newer versions)/absolute colorimetric)
Before proceeding you may want to use crop and rotate to straighten and crop the chart, as this will aid in the chart recognition process later on, leaving less room for failure.
Make really sure you have the basecurve and sharpening disabled (which are enabled by default). You’ll notice the image getting darker… Then do spot white balance on the target’s grey patches. Then use the color picker (bottom panel) to check the Lightness value (the L from LAB), of the brighest white patch. You need to check you’re charts reference file for the exact Lightness you should match. It’s usually L 96 for ColorCheckers, L 97 for most CMP Digital Targets and L 92 for most IT8 targets. If the Lightness of the brightest white patch is L 99 or L 100 straight away, the image is probably overexposed (and useless), move to an image shot a 1/3rd stop darker. Now open up the exposure plugin, and start increasing the exposure until the Lightness value of the white patch in the image matches the Lightness specified in your charts reference file. If you need to apply more than +0.3 or +0.4 of digital exposure, you probably need to check an image that was shot a 1/3rd stop brighter.
Once you find the proper image, and equalize the Lightness of the brightest white patch to the chart’s reference value, you need to inspect the image for glare, if there is any glare whatsoever you need to redo everything again. This is once again why I prefer matte targets, they’ll save you some aggravation…
If everything is dandy exit darkroom mode, and export a 16bit TIFF image.
Generating the color matrix using Argyll
To do the actual calculations we need Graeme Gill’s excellent ArgyllCMS, particularly version 1.1.1 or higher. First we need to read the TIFF file for it’s color patch values, we do this using Argyll’s command line utilities:
# scanin -v -p -a -dipn IMG_1234.tiff \
/usr/share/color/argyll/ref/CMP_Digital_Target-3.cht CMP_DT3.txt
This command will output a diag.tif which you can use to see if the chart was properly recognized. Using the previously read values we can calculate the actual matrix:
# colprof -v -A "Canon" -M "Canon EOS 400D DIGITAL" -D "canon eos 400d" \
-C "Copyright (c) 2010 Pascal de Bruijn. Some rights reserved." \
-q l -a m -u IMG_1234
This command will output a IMG_1234.icc file which is a standard ICC color profile. To directly use it with Darktable you can copy it for Darktable to find:
# sudo cp IMG_1234.icc /usr/share/darktable/color/in/canon_eos400d.icc
Submitting results
Since there is no direct way to check if a matrix’ values are correct, I’m hesitant to directly accept profiles/matrix values from third parties. At this point I highly prefer for users to make properly exposed RAW chart images available to us, so I can do the calculations. Proper credit will be given to whoever supplies the chart.
Screencast
The above video has been recorded at 1280×800, so it’s best viewed fullscreen. If you’re using Firefox to view the video you can just right-click and choose “Full Screen”. Or alternatively you can download the screencast here.