Darktable 1.0 Screencast Library (Addition)

Since I did my last darktable 0.9 screencast library, some things have changed. So at the very least this warranted an update screencast.

Darktable 1.0 Update (download)

Darktable Archiving & Backup (download)

These are the first screencasts that should be viewable on most tablet devices too, albeit with slightly degraded quality.

Darktable Unity Progress

Usually I don’t do a lot of “real” coding for Darktable, but I had some time on my hands today, and I implemented basic Unity integration for Darktable. Since I wasn’t familiar with libunity, nor was I really familiar with the depths of Darktable code let alone CMake, the implementation took me about 2 hours.

That said, have a look at the results:


You can also download the video for offline viewing if you prefer.

By the way, the bug I mentioned at the end of the video has been mitigated, which is a chic way of saying I kludged it so you won’t be bothered by it. But it’s not truely fixed.

Darktable 0.9 Screencast Library (Addition)

Since I did my last darktable 0.7 screencast library, some things have changed. So at the very least this warranted an update screencast.

The downside is that I was recovering from a cold, so these screencasts sound a bit rough:

Darktable 0.9 Update (download)

Darktable B&W Film Emulation (download)

Darktable Denoising (download)

Darktable Spot Removal (download)

Small mistake, it’s actually possible to remove spots by rightclicking them.

Darktable and Ubuntu Natty

For the past few weeks people have been bothering me about when I’ll update my PPAs for Natty… So let me explain my intentions for my PPAs in general.

In general

I have three Darktable related PPAs:

The difference between the Release and Release-Plus PPAs, is that the Release PPA only has the latest Darktable release package plus any dependencies that aren’t in the meanstream repo’s (lcms2 ATM). The Release-Plus PPA has everything the Release PPA has, plus updated versions of Lensfun and Exiv2.

For both the Release and Release-Plus PPA I’ll try to always build Darktable packages for at least the latest Ubuntu release and the latest Ubuntu LTS (Long Term Support) release. Please do note the usage of the word release, I usually do not build for unreleased versions of Ubuntu. And even after a release I need a couple of days to catch up.

The Unstable PPA is more or less my personal (but public) playground. It has a git master build of Darktable for whatever version of Ubuntu I’m personally using. Building packages for multiple versions of Ubuntu that often is just too time consuming. Of course people will say “Why not automate nightly builds?”. First off I think it’s a bit abusive of the PPA build farm, since git doesn’t change every single day. And more importantly there is no quality control on automated builds. I try to keep up on changes in git, and sometimes consciously refrain to push a new unstable package to my Unstable PPA, simply because there is too much risk of breakage. Obviously I do get things wrong from time to time, so my Unstable PPA can break things occasionally.

Now what about Natty

As I started out, I’ve been asked a lot about Natty, often in private e-mail which is uncalled for, since we have darktable mailing lists on SourceForge.

Now, there are some pretty big changes in Natty, and it’s quite debatable if they are for the better. So it’s very likely I’ll skip a release for my own use, which implies the Unstable PPA will most likely continue to be exclusively built for Maverick.

So that leaves my Release and Release-Plus PPA, and there has been build breakage on Natty, which only recently has been resolved. So it’ll involve backporting some patches to the current stable release to make Darktable build at all. I’ll certainly give this a shot once Natty has been released, and if all goes well I’ll quite likely have packages available within a week after Natty’s release. However, if building the current stable release on Natty proves too difficult I might ditch the effort altogether at some point, at least until Darktable 0.9 has been released.

Update

Darktable packages for Ubuntu Natty have been pushed to all three PPAs.

Darktable 0.7 Screencast Library

As some of you may or may not have noticed a new version of Darktable got released two weeks ago. And in the meanwhile I’ve been working on a set of somewhat shorter screencasts which should have good coverage of most features of Darktable and related topics.

So just in time for Sinterklaas I present to you, the Darktable 0.7 Screencast Library. I managed to improve the audio quality, though I sometimes make some silly mistakes. I call lighttable mode, lightroom mode here and there, and I seem to confuse left and right sometimes.

Darktable Installation On Ubuntu (download)

Darktable Lighttable Basics (download)

Darktable Tagging & Collect Plugins (download)

Darktable Darkroom Basics (download)

Darktable Darkroom Black & White Plugins (download)

Darktable Darkroom GND Plugin (download)

Darktable Darkroom Colorzones Plugin (download)

Darktable Darkroom Watermark Plugin (download)

Darktable Working With Styles (download)

Darktable Capture Basics (download)

The above videos have been recorded at 1280×800, so they’re best viewed fullscreen. If you’re using Firefox to view the videos you can just right-click and choose “Full Screen”.

Darktable Camera Color Profiling Screencast

I just suplemented my Darktable Camera Color Profiling article with a screencast:

The above video has been recorded at 1280×800, so it’s best viewed fullscreen. If you’re using Firefox to view the video you can just right-click and choose “Full Screen”. Or alternatively you can download the screencast here.

Darktable 0.6 Released!

After 5 month’s of hard work, we finally got around to releasing Darktable version 0.6. The new version has boatloads of new features. For example we added a lot of new image processing plugins. Other notable features are Picasaweb export and we can now storage password (like your Picasaweb login) in GNOME Keyring or KWallet (as any decent application should). Beside the new bells and whistles a lot of crashes have been fixed in the last month. Hats off to Johannes, Henrik, Tobias and the other contributers.

That said, my personal contributions to the latest release are mainly related to the basecurves… The basecurves allow us to have decent default output of camera RAW files. With proper basecurves Darktable’s output is roughly similar to the camera JPEG output. We now have basecurves for Canon, Nikon, Sony, Pentax, Olympus, Panasonic and Leica. We have a basecurve for Kodak as well, though that didn’t make it into the 0.6 release.

With the new 0.6 release I also took the oppertunity to cleanup my PPAs. I now have two seperate PPAs exclusively for Darktable. For “stable” releases add this PPA to your sources:

https://launchpad.net/~pmjdebruijn/+archive/darktable-release

Or if you’re feeling adventurous, and don’t mind the occasional problem, you can use our regular development snapshots:

https://launchpad.net/~pmjdebruijn/+archive/darktable-unstable

We’re now working toward a 0.6.1 release with amongst other things some user interface improvements.

Darktable Overview Screencast

I’ve been ranting a lot about Darktable lately, some of you might still have no clue what it’s all about. Also Darktable already has a lot of features, and like any complicated application some of it does need some explaining.

Screencasting has always been a great medium for users to learn about applications, it’s so much more efficient than documentation. Especially if we take a look at the huge success of Meet The GIMP, which was well earned.

I took a look at doing screencasts a while back, but back then (Karmic) recordmydesktop was semi-broken. On Lucid it seems to work pretty much perfectly, so I did a first attempt at doing a screencast for Darktable. At first this was intended as a testing-only video for me, but in the end turned out well enough to publish it. It’s still far from perfect, I recorded the audio with the amplification set slightly too high on an otherwise not so great microphone, so I had to do some postprocessing to clean it up (oggSplit, Audacity (Noise Removal & Compression), oggJoin).

The end result was a video that lasts for almost 50 minutes and covers pretty much all important aspects of Darktable, so it’s a great way to learn about Darktable if you can bear to listen to me for the duration of the video. For the short run this will most likely stay a one-off project. In the long run I’ll consider doing more shorter video’s detailing certain aspects of Darktable, taking a more structured approach to the content.

The video below is licensed under the terms of the Creative Commons Attribution ShareAlike 3.0 license.

The above video has been recorded at 1280×800, so it’s best viewed fullscreen. If you’re using Firefox to view the video you can just right-click and choose “Full Screen”. Or you can download the screencast here.

Darktable Camera Color Profiling

In the past I wrote about profiling for camera profiling for UFRaw. Since then I’ve pretty much switched to Darktable, and I’ve learned a few things. So here we go again, with a vengeance…

Why?

The pretty images our camera’s output aren’t literally what the camera sensors see, there is a lot of proprietary postprocessing involved. Now when working in the RAW format we ditch all that postprocessing in favor of RAW sensor data. Which is good if we want maximum control and flexibility. The problem is that camera vendors don’t publish their proprietary postprocessing methods, which leaves us with a problem, how do we postprocess then?

In the past some good work on this topic has been done by Adobe, who published their DNG specification including color matrices. Color matrices are specifications on how camera native color is transformed into something that an end user might like, and ideally will be correct when viewed on a calibrated display.

Historically there have been some problems with the Adobe color matrices, the biggest of these is the rendition of red colors. To this day I still have no clue why this really is a problem. The only explanation I can think of is that this has been a deliberate compromise on Adobe’s part to safely render skin tones, at the cost of red rendition elsewhere. This is speculation at best…

So ideally we want an alternate enhanced color matrix that will render red as it’s supposed to, possibly at the cost of some skin tones rendition (rendition of skin tones is often a matter of taste, so we can fix that using the color zones “natural skin tones” preset).

Types of profiles

Without going into all the nitty gritty details, there are basically two kinds of profiles, the first are XYZ matrices (these are often called color matrices) and are typically combined with a gamma curve, the second are LUT profiles. What’s a LUT, it’s a lookup table… The big difference is that with a XYZ matrix all color transformations are calculated on the fly, while LUTs are precalculated, so transforming color via a LUT is simply looking up an input color and it’s matching output color. The nice thing about LUTs is that they can deal with slight (nonlinear) deviations and can even be tweaked for creative purposes. So when generating a LUT profile, the profile is likely to pick up some of the peculiarities of your particular camera. XYZ matrices don’t have that problem since they are defined by only 3×3 coordinates in XYZ colorspace, and thus are quite generic by their very nature. This is also the reason why we are sticking to color matrices instead of supplying more detailed LUT profiles (besides diskspace usage).

What digital camera’s can be color profiled

All camera’s that can output a digital RAW format supported by RawSpeed can be properly profiled. This effectively means that most compact camera’s can not be profiled (yes, some could using the CHDK firmware hack). Probably half of all digital bridge camera’s can output RAW and thus can be profiled as well. Pretty much all digital SLR camera’s are covered as well…

What color target do I need?

There are currently lots of color targets available on the market produced using different techniques and sold at very different prices. Here are some of the things you need to keep in mind…

First we can classify all targets into two groups, matte and semigloss… In theory semigloss targets can cover a larger gamut (range of colors) than matte targets, but most semigloss targets tend to glare, which is a big inconvenience when shooting the target. Most IT8 targets are semigloss, CMP’s Digital Target for example is matte… I personally prefer matte targets for their convenience since they are easily shot using a decent strobe, which is often hard to do with semigloss target. Semigloss targets are often shot best outdoors on a sunny non-cloudy day at an angle to prevent glare.

Second there’s the patch count, Gretag’s Classic ColorChecker only has 24 patches, while most IT8 targets have up to 288 patches, and CMP’s Digital Target even has 570 patches. When one only want to generate a color matrix this is not of particularly great relevance, the ColorChecker’s 24 patches are enough. But if one wants to do LUT profiles more patches tends to be better.

Third there’s the production technique, some charts like CMP’s Digital Target are produced on highend inkjets, which implies the target is very vulnerable to moisture making it more fragile than other targets. Most IT8 targets seems to be produced using traditional minilabs, and last but not least some (typically Gretag’s) targets are painted. In theory painted target should be superior in the sense that the manufacturer has greatest spectral control over the patches, which means you’re less vulnerable to metamerism when shooting the target under different lighting conditions.

Fourth there’s reference measurements, some targets like Gretag’s ColorChecker are very accurately produced to a single reference specification.  Most IT8 targets are produced in batches, and will include per batch average reference data. The last and most accurate approach are individually measured charts, which means each chart has a reference which is specific to that chart alone, CMP does this for all it’s Digital Target’s and some IT8 vendors will also offer such services on request.

And last and least, don’t try to print your own target, even if you have a color calibrated printer, it will not be accurate enough!

Buying Advice

If you want a target that you can easily use with commercial software as well you should probably buy a (relatively expensive) ColorChecker Passport.

If you want a convenient and highly accurate target you should probably buy an (expensive) CMP Digital Target. This is my current personal preference.

If you want an affordable high quality target, and don’t mind some extra fiddling to get the shot right, you could probably buy an IT8 target from Wolf Faust.

Small note, Wolf Faust really kicks ass, since he relicensed his reference files under the terms of the Creative Commons Attribution-Share Alike license.

Shooting the color target

If you already have a target, or just purchased or borrowed one, we can try to shoot the target. First check if the target is clean and has no damage to the color patches. Then depending on the type of target you have available we can determine the shooting conditions:

Shooting a matte target

If you’re lucky enough to have a matte target available shooting it will be a breeze, since you can shoot the target head-on using a decent strobe. When possible put a dark grey or black piece of card behind the target. When I said decent strobe I meant something like a proper branded strobe like Canon, Nikon, Metz, etc. Please don’t do this with your generic asian no-brand flashes, since we don’t really know the quality of the light it produces, it might be fine for your photos, but I’d rather not use it for profiling purposes…

Shooting a semigloss target

If you have semigloss target the current recommended way to shoot it, is to wait for a sunny day with little or no clouds. Then usually at noon when the sun is highest in the sky shoot the target when there are no clouds obscuring the sun. When possible put a dark grey or black piece of card or cloth behind the target. You probably need to position the target at an angle (with regard to the camera’s position) to prevent any glare…

… Shoot the target using a modern normal lens (no antiques (since lens coating can differ) or fisheye folks) at about 50mm or so, at a distance of about roughly 1 meter or 3 feet. The target should cover about 50-75% of the image surface. The dark piece of board behind the target should be visible on all four sides of the target.

Now comes the hard part, getting the exposure right… Unless your camera has a true RAW histogram, the histogram will be useless to you. The best exposure will likely look slightly overexposed as a preview image. The best way to approach this is to take a boatload of exposures from severely underexposed to severely overexposed and select the best exposure later on using Darktable.

So, if applicable set your strobe to full manual (1/2 strength for GN36, 1/4 strength for GN48, 1/8 strength for higher GNs). Then set your camera to full manual mode as well, set Exposure to 1/200 sec, and set the sensor sensitivity to it’s normal lowest possible setting (without pushing), which is usually ISO 100 or ISO 200. Don’t use any special modes which allow extra low or extra high ISO, since this is often digital trickery… Then close down the aperture all the way (usually f32 or something) and take a shot, open up the aperture 1/3rd stop and take another stop, keep opening up the aperture and taking shots all the way, until the aperture is fully open. Usually this means you have taken about 20-30 shots by now.

Selecting and Processing a target image in Darktable

Now import the images you shot into Darktable, pick a target image which looks slightly overexposed, and open it in darkroom (develop) mode. Then disable all plugins except for these:

  • raw import
  • exposure
  • white balance
  • highlight reconstruction
  • input color profile (set to linear rgb (or rec709 in newer versions)/absolute colorimetric)
  • output color profile (set both to linear rgb (rec709 in newer versions)/absolute colorimetric)
  • display color profile (set both to linear rgb (rec709 in newer versions)/absolute colorimetric)

Before proceeding you may want to use crop and rotate to straighten and crop the chart, as this will aid in the chart recognition process later on, leaving less room for failure.

Make really sure you have the basecurve and sharpening disabled (which are enabled by default). You’ll notice the image getting darker… Then do spot white balance on the target’s grey patches. Then use the color picker (bottom panel) to check the Lightness value (the L from LAB), of the brighest white patch. You need to check you’re charts reference file for the exact Lightness you should match. It’s usually L 96 for ColorCheckers, L 97 for most CMP Digital Targets and L 92 for most IT8 targets. If the Lightness of the brightest white patch is L 99 or L 100 straight away, the image is probably overexposed (and useless), move to an image shot a 1/3rd stop darker. Now open up the exposure plugin, and start increasing the exposure until the Lightness value of the white patch in the image matches the Lightness specified in your charts reference file. If you need to apply more than +0.3 or +0.4 of digital exposure, you probably need to check an image that was shot a 1/3rd stop brighter.

Once you find the proper image, and equalize the Lightness of the brightest white patch to the chart’s reference value, you need to inspect the image for glare, if there is any glare whatsoever you need to redo everything again. This is once again why I prefer matte targets, they’ll save you some aggravation…

If everything is dandy exit darkroom mode, and export a 16bit TIFF image.

Generating the color matrix using Argyll

To do the actual calculations we need Graeme Gill’s excellent ArgyllCMS, particularly version 1.1.1 or higher. First we need to read the TIFF file for it’s color patch values, we do this using Argyll’s command line utilities:

# scanin -v -p -a -dipn IMG_1234.tiff \
      /usr/share/color/argyll/ref/CMP_Digital_Target-3.cht CMP_DT3.txt

This command will output a diag.tif which you can use to see if the chart was properly recognized. Using the previously read values we can calculate the actual matrix:

# colprof -v -A "Canon" -M "Canon EOS 400D DIGITAL" -D "canon eos 400d" \
        -C "Copyright (c) 2010 Pascal de Bruijn. Some rights reserved." \
                                                     -q l -a m -u IMG_1234

This command will output a IMG_1234.icc file which is a standard ICC color profile. To directly use it with Darktable you can copy it for Darktable to find:

# sudo cp IMG_1234.icc /usr/share/darktable/color/in/canon_eos400d.icc

Submitting results

Since there is no direct way to check if a matrix’ values are correct, I’m hesitant to directly accept profiles/matrix values from third parties. At this point I highly prefer for users to make properly exposed RAW chart images available to us, so I can do the calculations. Proper credit will be given to whoever supplies the chart.

Screencast


The above video has been recorded at 1280×800, so it’s best viewed fullscreen. If you’re using Firefox to view the video you can just right-click and choose “Full Screen”. Or alternatively you can download the screencast here.

Darktable

In the past I’ve always been a very assertive advocate of UFRaw (+F-Spot). I’ve never found rawstudio/rawtherapee any good for my purposes. However in recent months a new contender has emerged, and it’s called Darktable (obviously a pun on Lightroom (Lighttable & Darkroom)). It’s primarily authored by Johannes Hanika, who judging by the git log is a one man coding army.

There are a couple of things that interest me in Darktable, the first is that it is a combo of both photo management (lighttable) and raw development (darkroom), it’s really integrated, not two programs (F-Spot + UFRaw) stuck together with some ducktape.

The second would be the fact that Darktable seems extremely well designed, it’s entirely plugin based, even the raw import itself is a plugin. This means the code is well separated into these plugins, this inherently means functionality can be disabled by just disabling a plugin. No need to comment out some code before compiling.

Another strong point of Darktable is it’s imaging pipeline, except for the first and last few steps, the pipeline is entirely in 32bit floating point LAB colorspace, properly integrating color management. Which means it’s calculations can be extremely accurate. A nice extra benefit is the fact that some image operations just work better in LAB than other colorspaces, the Unsharp Mask being the principal example of this.

Something that’s also very important to me is the user interface, I really care about this. While Darktable does not always conform to common practice in user interfaces and often prevents novel idea’s for it’s user interface, it’s easy to work with. It works particularly well fullscreen (F11 just like Firefox), it’s dark theme makes sure you’re not distracted from your photograph. And the edge/buttons for expanding or collapsing the sidebars are at the screen’s edges, which means you can just quickly drag your mouse to the edge (without any particularly aim), and click to expand or collapse a sidebar, this works really fast!

Darktable is still a very young project, but it’s well worth taking a look at. Rolf Steinort from Meet The GIMP recently did a show on Darktable.

If you want to try Darktable yourself can get nicely packaged git checkouts (development versions) from my PPA for Ubuntu.