Metadata Madness

Dirty Rat Bastards

2011-03-31T12:31:00.000-07:00

Samsung is playing dirty with the metadata of its NX10 camera. While other manufacturers like Nikon and Sony encrypt colour balance information in RAW images (presumably to prevent 3rd party RAW converters from using this information), Samsung has taken this to another level with the NX10 (and various other recent models):

The difference is that the stored information is actually usable, but is changed just a little bit by adding small, random encrypted amounts to the stored values. The effect would be that the images wouldn't be as good from 3rd party utilities using this information.

Nasty.

But the next version of ExifTool (8.54) will remove this obfuscation from the stored values.

Chalk up one for the good guys.

More Windows corruption woes

2010-03-26T11:49:00.000-07:00

Yet again I get a question from a user with images that have been corrupted by editing metadata with Windows. The biggest problem is that Windows may change the byte order of the EXIF when the metadata was edited. This is a REALLY, REALLY BAD thing to do because it is impossible to properly change the byte order of some proprietary information. So the effect is that this information is corrupted, causing errors when other software that parses this information reads the image.

This was a problem with Windows XP up to and including Service Pack 3, but from one report this seems to have been fixed in Vista and Windows 7. If so, there must still be a lot of people running XP out there because I keep seeing this complaint.

First confirmed XMP in original images

2010-03-24T04:56:00.000-07:00

I think I now have enough samples to conclude that Canon is embedding XMP into images directly from the camera. Currently the XMP contains only a single tag, "Rating", which has a value of 0 in all my samples. While I thought there was a chance that all of my samples had been modified by some Canon software utility which added the XMP, I have now obtained enough samples from different sources that I believe this is not the case. The samples I have with XMP are from the following PowerShot models: A3100IS, SD980IS, SD1400IS, SD3500IS and SX210IS.

This is significant because in the past the limitations of EXIF have forced camera manufacturers to store information in a proprietary format in images, limiting its usefulness. But now that manufacturers are embracing XMP there is a chance that some of this information could be put in a more useful format.

For instance, it would be GREAT if cameras like the Panasonic DMC-ZS7 stored its location information (Country, City, State, etc) in XMP. Currently this information is stored in the maker notes, which means it is unavailable to most software. (The only reasonable option with the maker note solution is to use ExifTool to copy the information to XMP, which is one of the very useful functions of ExifTool, but in an ideal world this would not be necessary.)

More web server stats

2010-03-22T10:16:00.000-07:00

I spent some more time watching the ExifTool web server to see what was getting the most traffic. The most popular file is the "ExifTool Updates" RSS feed, which gets an average of about 2.3 hits per minute -- that's 100,000 hits per month (from more than 20,000 unique IP's). Due to the nature of RSS, I was worried about this file dominating our bandwidth, so I tried to keep it small (12 kB) and set its reload period to 90 minutes (for RSS readers that honour the TTL attribute). Good thing I did that.

ExifTool download statistics

2010-03-16T06:49:00.000-07:00

I just checked the web server to see how many people are downloading exiftool. I am now getting more than 1000 downloads per day from the main server. This doesn't include all of the downloads from the various CPAN mirrors and other download sources (which all together could easily be another 1000 per day).

At this rate I could retire with a $40,000 per year salary if I just charged $0.05 per download.

But ExifTool is free.

Stupid Samsung SRW

2010-03-01T05:14:00.001-08:00

Samsung has invented another TIFF-based RAW format called SRW (used by the NX10).

The idiot Samsung engineers haven't included any sort of file identifier to allow this type of file to be distinguised from a TIFF image. This means that software must rely on the file extension to be able to read or write this file properly. This really pisses me off. Will they never learn?!

Also, they have included no identifier in their maker notes, which again is not a good idea.

The best one can do to identify these images is to look for a SubIFD1 Compression value of 32770, but this can only be found after processing IFD0 and the SubIFD's.

It would be much better if the file included a signature after the TIFF header such the Canon CR2, or a different TIFF magic number like Olympus ORF.

[Edit - discovered more idiocy]

Another BLACK MARK against the Samsung engineers:

The SRW format stores the ThumbnailImage inside a SubIFD of IFD1 instead of directly in IFD1 as per the EXIF specification.

Why are they so stupid?

Production Leica S2

2010-02-22T11:03:00.000-08:00

Well, support for the insane Leica S2 maker notes was finally added in the version of ExifTool that was released a few days ago. Adding this support was as painful as I had anticipated. As it turns out, images from the production model have all the same problems of the pre-production samples. The question about the funny 0x800 offset is explained by the fact that a different offset base is used for one maker note entry (coupled with lots of unnecessary padding to keep the value at a constant offset). Anyone who understands the TIFF IFD structure will realize how stupid this is -- one may draw their own conclusions about the level of understanding by Leica engineers.

Leica S2 Fiasco

2009-11-19T10:24:00.000-08:00

Yet again the JPEG 64kB segment-size limitations make life difficult for me.

Presumably to avoid this limitation, the new Leica S2 writes the complete maker notes (860kB of them) outside the EXIF segment, and instead writes them after the JPEG EOI. Needless to say, this complicates things for me yet again.

But there are other problems in the S2 maker notes that make me pray my samples are from a pre-production camera and that the format will be fixed in the production model. Many maker note values, including one offset, have a constant value of 0x0800. This can't be right. I will wait until I get some production samples before adding support for this camera.

Software designed to make you feel stupid

2009-11-05T09:11:00.001-08:00

Adobe LightRoom

The first time I tried to use Adobe LightRoom to edit metadata was frustrating to say the least. First I had to figure out how to import the image (why? the image was right there on my desktop!!), then I had to find the metadata panel (which is quite a challenge since it had scrolled down and out of view in one of the panes to the right). I didn't find this panel at first, so the "Metadata" menu entry drew my attention. Sounds good. That's what I want, right? However, this menu seems to deal only with a group of metadata presets, which complicated things since I just wanted to change a single picture. Oh well. Forge ahead. Editing the metadata presets seemed like the right thing to do. When I was done editing LR forced me to save the presets with a new name, which I didn't want to do, but I didn't have any choice, so I did it anyway. Then, the next obvious step was "Save Metadata to File", which didn't work. Darn. Presumably it didn't work because I needed to somehow select the presets that I had just saved, but how? I could see no entry in the menus to set the current metadata presets. At this point I seem to remember taking a time out and coming back to the problem another day. Eventually I was able to somehow change the presets and write my changes to the file, but right now for the life of me I can't figure out how I did this (I have LR open now and I'm trying to reproduce what I did). Oh wait. There it is. Right-click on the image and select the presets from the "Metadata Presets" menu item (the good ol' hidden menu to the rescue). No, wait. That didn't work. Try a splat-S for good luck. Hooray! Success! Finally.

Capture One

Yesterday I downloaded a copy of Capture One and tried to edit some metadata in an EIP (Capture One Enhanced Image Package RAW image). From my experience with LR, I guessed I would first have to figure out how to import the image. So I did this, which was confusing enough, but then for the life of me I couldn't see how to select the image once I had imported it. You would think it would be displayed somewhere in the library of images, but no. I couldn't find it. Finally after some searching on the hard disk I found it nested deep below a "Capture One" directory in my "Pictures" folder. OK, cool. Select the image in the browser pane from that location on the disk. At least the metadata pane is not hidden, so I click on an IPTC entry and try to change it. Nothing. Can't enter any text. Darn. Time to RTFM. The FM says that I must 1) import the image. Good. Done. Let's proceed. Then 2) click on the IPTC entry. Good. Done. What next? 3) Type in new value. Darn. Bummer. Tried that. The keyboard focus never changed. The nice picture in the FM shows a highlighted text entry field. Can't make that happen. Clicking a thousand times still doesn't bring it up. Try going back and re-importing different images to different locations. Now I have copies of images scattered across my hard disk and I still can't change any metadata. After about an hour of searching through the menus, clicking all the various tiny icons in the interface, and RTFM-ing, I finally give up. It just isn't worth it. I have to watch my blood pressure.

Sweet Irony

...and people complain about the ExifTool command line interface.

Date/Time tags

2009-10-23T09:04:00.000-07:00

A scenario: On 2009/10/23 I edited an image that I had originally scanned from a photograph on 2009/09/12. The date stamped on the back of the photograph was 2009/05/16 (the date the film was processed). The picture was taken in a museum on 2009/05/01 during my trip to Washington DC. The subject of the picture is a 2005 painting by Tom Freeman which depicts the bombing of Pearl Harbor on 1941/12/07.

The question: For this image, what should be the values of each of
these tags? (The MWG tag descriptions are in brackets.)

ModifyDate - ("modification date of the digital image file")
CreateDate - ("creation date of the digital representation")
DateTimeOriginal - ("creation date of the intellectual content being shown")

The answer?:

DateTimeOriginal - I think it is fairly clear that this should be 2009/10/23, the date that the image was last modified.
CreateDate - If we take the MWG description literally this should be 2009/09/12 (the date that the image was scanned), however this date is perhaps not as meaningful as some of the other possibilities.
DateTimeOriginal - This one is up for grabs. Any of 2009/05/01, 2005 or 1941/12/07 could be possible.

DateTimeOriginal is the most difficult. What is meant by "intellectual content being shown"? For any other general picture I took during my trip, this would be easy, and it would be 2009/05/01, the date of my trip. However, the subject of the picture is a painting which was created in 2005. But the content of the picture is the 1941/12/07 raid on Pearl Harbor.

I suppose the proper value for DateTimeOriginal depends on the context in which the image is used. If I want to associate the image with other pictures from my trip, then 2009/05/01 is the most appropriate date to use. However, if I am using it to document an essay on American history, then 1941/12/07 is more significant, or if used in a biography of Tom Freeman, 2005 is the appropriate date.

Confusing.

Déjà vu

2009-06-23T10:06:00.000-07:00

CIPA has recently released a "Multi-Picture Format" (MPF) standard for storing large images in JPEG files. Again, there is a big problem with this standard: It uses offsets that are relative to the start of the MPF header (in the new MPF APP2 segment) to reference images after the JPEG EOI. These offsets will quickly be broken if any data after the MPF segment changes length. This problem could have been avoided if offsets had been specified relative to the end of file, but it is too late for this now that the specification is public.

The only workable alternative I can see is to enforce the rule that the MPF APP2 segment must come after all other APP segments. (It would have been smart if this was specified in the CIPA standard, but sadly this isn't the case.) If this is done, then metadata in the remaining APP segments (EXIF, IPTC, XMP, etc) can safely be edited without breaking the MPF offsets. I suggest that all metadata editors employ this strategy, regardless of the segment order specified in the standard (which says that the MPF APP2 segment must come immediately after the EXIF APP1 segment).

Yet another PreviewImage strategy

2009-05-26T08:50:00.000-07:00

One big (huge, actually) problem with JPEG images is that they are limited to segment sizes of less than 64 kB. Since the EXIF information must go in one segment, this forces camera manufacturers to invent their own ways of storing larger data blocks. This is a real problem for the preview image, which many manufacturers write in their JPEG's, and which can easily push the EXIF data size to over 64 kB.

The problem is that every manufacturer is using a different technique to store the image.

Canon, Olympus, Konica/Minolta and Sony cameras write the PreviewImage after the JPEG EOI. This technique allows a contiguous preview to be stored, but trailers like this are typically lost if the image is edited. So this solution is not ideal.

Nikon, Pentax and Casio keep the PreviewImage small enough to fit inside the EXIF APP1 segment (less than about 30 kb), which makes the images too small so they are only useful as a thumbnail.

Kodak writes the image to a special APP2 FPXR segment, which is actually part of the EXIF specification, but the format is a Microsoft-devised abomination that nobody in their right mind would ever think of using. Oh, except FujiFilm, who write the image in this segment, but don't bother to write all the necessary table of contents to be able to read it using the standard technique.

I have just discovered that the new Samsung cameras recently started embedding preview images larger than 64 kB, and of course they created a new technique to do so. If they were smart, they would have developed a simple technique that could be used by others in the future, but of course they were stupid, and didn't think that far ahead. (Such is the normal path of dumb camera manufacturers when it comes to metadata.)

The new Samsung models simply split the preview and write it to separate APP2 segments with no header. If they had written a header (like "PREVIEW\0" for example), then the technique could be portable and useful. But they didn't. Without a header, the data can not easily be distinguised from other random APP2 data, so this technique is not generally useful.

Why not just use APP2 with a simple "PREVIEW\0" header? If everyone did this, life would be much simpler.