It's a special function of the tfm 2-pass file, it errors out if you don't use exactly the same filters before tfm as you did on the first pass. tdec has the same check. Colormatrix should have no effect, but tfm's trying to protect against random changes that render the stats useless.
To disable the check, all you have to do is remove the "crc=" section from the beginning of the stats files. (Leave the other info though!)
Colormatrix will still have an effect where it is now, though. It just won't be able to use hints to decide when it actually needs to be used, it'll just affect every frame (which is usually correct).
URL:
http://forum.doom9.org/showthread.php?p=822415#post822415