Ownership of RAW file formats

[Bruce McL]

Hello there,

As a long time computer professional I have some concerns about some RAW formats becoming obsolete over time. This happens far to often with writers who can no longer read their documents because they are on floppy disks made for a Kaypro or were written in BeagleWorks and they don't have a copy of it anymore. I'd hate to see this happen to photographers. Does committing to RAW mean committing to owning a 2005 era computer for the rest of your life?

Maybe you can help me understand this better. Who owns the various RAW file formats? Are format specifications available to the general public or are they private and proprietary? To me it seems like having a camera that has a RAW file format spec in the public domain is a very good thing. Is anybody else even concerned about this?

[MichaelT]

The Kaypro analogy is not quoite the same because of the massive quantities of camera being sold today and in ownership of th pubic. So i for one am not worried that my D30 RAW files will not be able to be converted in the future. If I owned a Pentax *ist I might be more concerned.

However your point is valid.

Adobe has put forth an intiative that address this called DNG (Digital negative). Uou can read about it on their site and on Luminous-landscape.com. i will have ore to say about it on this site as well. Overall it is a good thing, but not a prefect archival format.

I for one am not going to worry about this as any image that is worthwhile is already stored both in its RAW format and in a "print" configuration with sharpening applied. So i will always have the Print format, and i think that they RAW will also be accessable in the future.
Also as the MP war soon ends (within a few years), I think that generic Nikon and Canon formats will start to be come consitiant between camera models.

Just my 2 cents. YMMV.

[Bruce McL]

Thanks for your reply. I'm glad you understand my concerns, and yes I was a bit overly dramatic with my Kaypro analogy!

Interesting comment about the Pentax *ist. I agree. What if Pentax gets bought by a rival in a few years?

I have read a little about DNG and it does sound promising for the future. Maybe when DNG comes out it will be possible to convert RAW files being produced with current cameras into DNG format. I think that would be much better for long term storage.

To sum up I would like to use the analogy of developing film. Analog film developing is an open standard, the tools and techniques are well known and publicly available. I think that by using RAW as it is now the photographer is ceding some control of the "digital film developing" process to the camera manufacturers. It would not be difficult or expensive for camera manufacturers to put their file specs in the public domain. It would be a service to the photographic community. If done right it could even make RAW file support easier and less expensive for manufacturers.

Thanks for listening, -Bruce.

[skirkpatrick]

I'd be interested to hear Michael's take on this (which he has been promising to put in print for some time now).

I'm not too worried about my Olympus E-1 .orf files being unreadable in the future as the image data itself is not compressed or encrypted (these are essentially the same thing) and is apparently very easy to locate in the file. But getting a good image out of it also requires reading the meta data and Olympus (like all the others) stuffs the majority of these items into the "EXIF MakerNotes" fields, which are not documented or standardized. As I understand it, the .dng format just passes these along, but does not attempt to move them into standard form. Thanks to a recent thread on the dpreview Olympus DSLR forum, and some experimention by people willing to read RAW data files in an editor, there is a public domain tool to read things like the camera serial number, the focus distance and other things that Olympus has not chosen to expose in their own software. See

http://www.sno.phy.queensu.ca/~phil/exiftool/

for the result. But it is not for the faint-hearted. This is a Perl script. Windows users have to install Perl and then run it from the command line. It is only a starting point that a vendor like CO can use before testing to see if the tricks it uses to extract stuff are generally reliable. And while ExifTool can read my Olympus metadata with some success in an .orf file, I don't know if it could do the same after the makerNotes have been dropped into a .dng file.

I think the requirement to redevelop a favorite image for new output media at some unknown future time is not met by keeping a Tiff or Jpeg around with a single sharpening setting applied. Since profiles change from time to time, this is the reason I keep asking CO to label each output file with the version of CO that it was made with. This would seem to fit in the "software" tag, but CO is already using that one for the software used in capturing tethered from their medium format backs. Until they find another place to put this information, I will be able to tell my jpegs apart by using other EXIF bugs as fingerprints.

[MichaelT]

I think that the real irony is that the camera manufacturers do NOTHING to help 3rdd party developers, even though they are dependant on us to sell their cameras. In the case of Canon they do it because they want the upper hand in not etting out their flaws (my opinion), in the case of Nikon it is to protect their Nikon Capture profit center. In the case of all Japaneese mfgs, it is simply that that is the way of the Japaneese mind set.

The bottom line os that these companies are not working in the best interests of their customers (what else is new). That is the real crime.

DNG is helpful, but not the final solution. Like I said using the *ist example, I would be concerned if I has an :obsure: camera, but being a canon shooter, I am not afraind that I will not be able to process my files in Capture One or Capture Ten or Adobe CS36 ($4500).

YMMV

[PaulT]

You can see Chuck Westfalls' initial response to someone looking for file format documentation here: http://www.figuiere.net/hub/blog/?2005/06/04/201-canon-and-raw-file-official-statement

It's unfortunate but this kind of attitude is not limited to Canon, or even 'comsumer' devices (not that I think a 1D or 1Ds is really consumer). The team behind OpenBSD, an operating system like Linux, have had, and continue to fight against, problems like this from all manor of hardware manufacturers. See http://marc.theaimsgroup.com/?l=openbsd-misc&m=111118558813932&w=2

In the end it comes down to the company wanting to hold onto some market share or dominance, which I think flies in the face of consumer choice and a free market. It's not all about the 'opensource' operating systems but rather the ability of the consumer to make use of their property as they see fit. Chuck mentioned in his response linked above that Canon feels that their posture in no way affects the photogs copyright claims, which is indeed true. I'd suggest however that copyright is a somewhat hollow right if you are restricted from using it in the way you desire.

Microsoft including support for, and getting software or docs from, camera manufaturers for the upcoming Longhorn will only make it more difficult to get past the marketing. It's also highly likely that Canon et al will attempt to justify their position by saying '90% of desktop comuters have support by default'.

It's a sad state of affairs which seems to have common threads in everything from software patents to privacy laws. The interests of [multinational] commerce are now pushed so hard that the interests of the public or small enterprise are overlooked or marginalised out of existence.

Nothing really new said here but if you're interested visit http://www.openraw.org/

[MichaelT]

RawWorkflow.com and Pixmantec ApS, as well as me personally, all support the OpenRaw initiative. www.openraw.org

OT: What I do not support is someone referring to the Canon people as "dicks" as was done in one of the links sited. Chuck Westfall is a wonderful human being doing his job, regardless of the position of canon that he must follow, or his personal view. We only diminish any position when we resort to personal attacks.

[PaulT]

What I do not support is someone referring to the Canon people as "dicks"

I can't agree more. I've never personally dealt with Chuck but have read numerous postings from him as a representative of Canon and have always found it useful to have an active member of the online forums from Canon. I'm sure most people reading this forum would realise that I'd intended the link to point to the quoted response and any comments found at that page are the views of their respective posters, not necessarily myself, that site maintainer, this site maintainer or anyone else on Earth. I can't speak for those from Mars.

Back to the topic:
I wonder how it will all pan out. Will another defacto standard get created in the same way PNG was created to usurp JPEG, OggVorbis for MP3 and the like, or will the manufacturers concede to what is in, I think is, the public interest?

If there really is intellectual property embedded into the file format it would, I think, suggest a poor implementation. We are after all talking purely about 'data'. How that data was captured, calculated and manipulated into it's stored form should really not be necessary in the file. If their [processing] software does custom processing on that data to get the best image from the RAW then fine, and I can understand not publicising that info. Likewise for any processing done in-camera. But it's not the process[ing] that is being sought, merely the end result.

Maybe I just don't understand enough about what is currently stored in the RAW files, but maybe that's just the point. What may be currently stored there and what should be stored there, according to my thoughts, may not be the same thing.

[MichaelT]

Paul,

Dic...
I knew that YOU did not make those statements or endorse them.

Regarding the RAW. 1 valid point of the mfg is that as long as people do not have the knowledge to create a Canon(for example) RAW file, Law enforcement can be better ensiured that a RAW image is an original and not manipulated. But that is 1 point.

regarding how the data was stored as opposed to the data itself, I think you are right. Even Fuji with their spcial 2 part pixel sites, only have to tell us where the data is, not how it got there or how to use it from there.

Let's support www.openraw.org to best leverage out position and future security of our files.

[PaulT]

1 valid point of the mfg is that as long as people do not have the knowledge to create a Canon(for example) RAW file, Law enforcement can be better ensiured that a RAW image is an original and not manipulated. But that is 1 point.

That's really quite scary. Security through obscurity, then applied to an environment where someone may loose their freedom [prison], life [execution], or anything in between [fine, criminal standing, ...]. The fact that law enforcement agencies may (I don't know if they do) presume that an image is original based partly on the fact that the mnfr doesn't release documentation on how the image is formatted would indicate an implicit trust. What they should, I feel, be requiring is an explicit guarentee, something like cryptographic signatures, with some mathematically calculable probability of being forged.

Isn't there currently a spate of cases in the US where people charged with DUI are being released because the mnfr of the eqpt is not willing to release information about how the test is performed? It's perhaps a little different but if originality is actually only implied by hiding the format, history shows that at some point it will be reversed, and likely long before the agencies/courts are aware of this.

Added: Canon et al are of course not required to implement anything. They do not sell their products as fit for any purpose, the fact that they are used in any particular environment is up to those involved. One would assume however that if law enforcement or the courts required that originality be verifiable then they would be remis for not taking advantage of the market opportunity.

[MichaelT]

Canon does sell a kit that verifies the authenticity of RAW image files, so I may have been out of line with my using law as a reason for canon for securing the format. In any case *I* support OpenRAW and believe that all RAW data should be documented so that we can do what we want with out files.

Just a thought...does MS document the MS Word or Excel format so that we can have access to our documents 50 years from now?

[PaulT]

Just a thought...does MS document the MS Word or Excel format so that we can have access to our documents 50 years from now?

Funny that, they didn't but now have decided to do just that. Maybe a result of the EU case or some other motive, but I believe that from the next version of Office they will be using a publicly documented XML format. This is probably a more globally important example of the same issue, and if MS concede the point, maybe Canon et al can/will too.

Totally off topic: Does Phase One document their work file format? Does C1LE 3.7 import C1LE 1.2 work files? [My tests suggest not]. I've kept a copy of 3.5.2 installed which seems to handle the 1.2 files OK.

[MichaelT]

V3.7 was conceived to import all old work files, but i do not know if they have suceeded. I have moved on from Caoture One as you know..

...and if MS is going to an XML compatable format, then that is good for all of us. I always thought that the Autocad .DXF interchange was a great model. They did not have to give up their secrets, but anyone could read their files. That is all we are asking of the camera mfgs.

[JohnMalloy]

Totally off topic: Does Phase One document their work file format? Does C1LE 3.7 import C1LE 1.2 work files? [My tests suggest not]. I've kept a copy of 3.5.2 installed which seems to handle the 1.2 files OK.

Not totally off topic at all, you do have to move your workfiles through the various upgrades to make sure they are current. They do have to be updated by 3.x before they are treated properly by 3.7...so they will work, but not directly.

FWIW, I use COne as a RAW converter, so backward compatibility takes a back seat to providing the best possible conversion available today. If I can archive a RAW file, I can certainly archive it's present finished form (tif or jpeg) just as well. That's what I do to avoid the possibility that it may not be convertable in the future...

The writer who chose to use a computer/software without hard copy or without following proper upgrade paths was remiss. He'll definitely also have problems replacing his 8 track in his '51 Studebaker when he gets that running again....

This is not sarcasm...I do belive in open source, however, I also understand the advantages and necessities of a competitive marketplace for developers.