[Flac] 24 bit question

[scott brown]

Someone sent me a question late last night and I briefly looked at his file
this morning and couldn't figure out the answer, so I'm posting here.

A friend has a a ~275MB 24 bit, 48khz stereo wav file of rock music that
when compressed using flac level 8 gives a flac file under 110 MB in size.
When I dithered his file to 16/48 and converted that file to flac, the
resulting flac file was actually 2 MB *bigger* than the corresponding 24/48
flac file. Does this make sense to anyone?

He says that his 24/48 files always compress to around the same size as the
same files converted to 16/48 or 16/44.1. I couldn't give him an answer as
to why.

Does anyone have an answer?

Thanks,

Scott
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.xiph.org/pipermail/flac/attachments/20101202/e0518699/attachment.htm

[Nicholas Wilson]

24bits are only 14% more information than 16bits, not as much as it
looks. I presume the downsampling introduces some noise which
compresses poorly (bigger residuals) and pretty much outweighs the
advantage. This does not happen with a lossy codec, if the same error
tolerance is imposed on each stage: a 5% noise introduction (say) at
one stage does not create a problem if a 5% error is allowed to be
introduced later to discard the noisiest 5% of the data. So,
heuristically I would expect most all the gains of reduced detail to
be realised in lossy codecs, and rather little or no space saving with
lossless codecs.

Nicholas

-----
Nicholas Wilson: nicholas at nicholaswilson.me.uk (ncw33)
Site and blog: www.nicholaswilson.me.uk
Peterhouse, CB2?1RD ? 86 Heath Road, GU31?4EL

Post by scott brown
Someone sent me a question late last night and I briefly looked at his file
this morning and couldn't figure out the answer, so I'm posting here.
A friend has a a ~275MB 24 bit, 48khz stereo wav file of rock music that
when compressed using flac level 8 gives a flac file under 110 MB in size.
When I dithered his file to 16/48 and converted that file to flac, the
resulting flac file was actually 2 MB *bigger* than the corresponding 24/48
flac file.? Does this make sense to anyone?
He says that his 24/48 files always compress to around the same size as the
same files converted to 16/48 or 16/44.1.? I couldn't give him an answer as
to why.
Does anyone have an answer?
Thanks,
Scott
_______________________________________________
Flac mailing list
Flac at xiph.org
http://lists.xiph.org/mailman/listinfo/flac

[Brian Willoughby]

Nicholas is probably right about noise. Another factor would simply
be the amplitude of the resulting file.

A 275 MB 24-bit file which compresses to 110 MB is probably not very
loud. I assume that the average level is somewhat low, with few if
any peaks that reach 0 dBFS. FLAC is very good at compressing audio
that is not loud. In fact, the quieter the recording, the smaller
the FLAC file. I've actually mastered some audio where the final 24-
bit FLAC was larger than the original 24-bit FLAC, simply because the
amplitude is larger on average after mastering.

When dithered to 16-bit, I'd assume that your WAV output is about 183
MB, is that right? Seems like you're saying that this only
compresses to 112 MB. That is consistent with a louder file and/or
one with more noise. I assume that you may have boosted the level of
the recording before dithering to 16-bit, or maybe even just
normalized the 24-bit file. Audio dynamics compression is expected
to increase the size of the FLAC output. If you did not alter the
volume at all before dithering, then I suppose it's still possible
for the dither noise alone to reduce the efficiency of the FLAC
compression. What was your dithering process?

In general, FLAC compression achieves about 50% reduction in size. I
get better than 50% with raw 24-bit recordings, and worse than 50%
with mastered 16-bit recordings. In your case, the 24-bit file is
compressed to 40% of the original size, but the 16-bit is only
compressed to 61% of the 16-bit WAV. These values are all consistent
with my typical experience, especially since there is an incentive to
make the 16-bit version louder to avoid excessive quantization noise.

Brian Willoughby
Sound Consulting

P.S. Nicholas, where did you come up with that 14% figure? A 16-bit
file is 33% smaller than a 24-bit file, or a 24-bit file is 50%
larger than a 16-bit file. I can't find any math that works out to a
mere 14%.

Post by Nicholas Wilson
24bits are only 14% more information than 16bits, not as much as it
looks. I presume the downsampling introduces some noise which
compresses poorly (bigger residuals) and pretty much outweighs the
advantage. This does not happen with a lossy codec, if the same error
tolerance is imposed on each stage: a 5% noise introduction (say) at
one stage does not create a problem if a 5% error is allowed to be
introduced later to discard the noisiest 5% of the data. So,
heuristically I would expect most all the gains of reduced detail to
be realised in lossy codecs, and rather little or no space saving with
lossless codecs.

Post by scott brown
Someone sent me a question late last night and I briefly looked at his file
this morning and couldn't figure out the answer, so I'm posting here.
A friend has a a ~275MB 24 bit, 48khz stereo wav file of rock
music that
when compressed using flac level 8 gives a flac file under 110 MB in size.
When I dithered his file to 16/48 and converted that file to flac, the
resulting flac file was actually 2 MB *bigger* than the
corresponding 24/48
flac file. Does this make sense to anyone?
He says that his 24/48 files always compress to around the same size as the
same files converted to 16/48 or 16/44.1. I couldn't give him an answer as
to why.
Does anyone have an answer?

[scott brown]

Thanks for the replies!

My first thought was that the file had low levels (before he sent me the
file), but that's definitely not the case with this file. There are many
peaks that reach 0dBFS.

He sent me the original wav this morning and I loaded it into Wave Editor on
OS X. I dithered to 16 bit using MBIT+ (high/ultra setting) and saved the
16 bit file. I did nothing else (no normalizing or any other processing).
I can't give you the 16 bit size right now since I'm at work and the file is
on my Mac at home, but I can report back tonight.

Whatever my process is, though, the guy who originally recorded the file
gets the same results with whatever method he uses to convert to 16 bit on
Windows. I can ask him what his 24 > 16 bit process is. I can also just
truncate the file down to 16 bit and report back on the resulting flac file
size. Would you expect that flac file to be around the same size as the 24
bit? In my experience, my 24/48 flac files are always substantialy bigger
than my 16 bit flac files, which is why this case confuses me...

Thanks,

Scott

Nicholas is probably right about noise. Another factor would simply be the
amplitude of the resulting file.
A 275 MB 24-bit file which compresses to 110 MB is probably not very loud.
I assume that the average level is somewhat low, with few if any peaks that
reach 0 dBFS. FLAC is very good at compressing audio that is not loud. In
fact, the quieter the recording, the smaller the FLAC file. I've actually
mastered some audio where the final 24-bit FLAC was larger than the original
24-bit FLAC, simply because the amplitude is larger on average after
mastering.
When dithered to 16-bit, I'd assume that your WAV output is about 183 MB,
is that right? Seems like you're saying that this only compresses to 112
MB. That is consistent with a louder file and/or one with more noise. I
assume that you may have boosted the level of the recording before dithering
to 16-bit, or maybe even just normalized the 24-bit file. Audio dynamics
compression is expected to increase the size of the FLAC output. If you did
not alter the volume at all before dithering, then I suppose it's still
possible for the dither noise alone to reduce the efficiency of the FLAC
compression. What was your dithering process?
In general, FLAC compression achieves about 50% reduction in size. I get
better than 50% with raw 24-bit recordings, and worse than 50% with mastered
16-bit recordings. In your case, the 24-bit file is compressed to 40% of
the original size, but the 16-bit is only compressed to 61% of the 16-bit
WAV. These values are all consistent with my typical experience, especially
since there is an incentive to make the 16-bit version louder to avoid
excessive quantization noise.
Brian Willoughby
Sound Consulting
P.S. Nicholas, where did you come up with that 14% figure? A 16-bit file
is 33% smaller than a 24-bit file, or a 24-bit file is 50% larger than a
16-bit file. I can't find any math that works out to a mere 14%.

Post by Nicholas Wilson
24bits are only 14% more information than 16bits, not as much as it
looks. I presume the downsampling introduces some noise which
compresses poorly (bigger residuals) and pretty much outweighs the
advantage. This does not happen with a lossy codec, if the same error
tolerance is imposed on each stage: a 5% noise introduction (say) at
one stage does not create a problem if a 5% error is allowed to be
introduced later to discard the noisiest 5% of the data. So,
heuristically I would expect most all the gains of reduced detail to
be realised in lossy codecs, and rather little or no space saving with
lossless codecs.

Post by scott brown
Someone sent me a question late last night and I briefly looked at his file
this morning and couldn't figure out the answer, so I'm posting here.
A friend has a a ~275MB 24 bit, 48khz stereo wav file of rock music that
when compressed using flac level 8 gives a flac file under 110 MB in size.
When I dithered his file to 16/48 and converted that file to flac, the
resulting flac file was actually 2 MB *bigger* than the corresponding 24/48
flac file. Does this make sense to anyone?
He says that his 24/48 files always compress to around the same size as the
same files converted to 16/48 or 16/44.1. I couldn't give him an answer as
to why.
Does anyone have an answer?

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.xiph.org/pipermail/flac/attachments/20101202/36b391d4/attachment.htm

[Brian Willoughby]

Post by scott brown
My first thought was that the file had low levels (before he sent
me the file), but that's definitely not the case with this file.
There are many peaks that reach 0dBFS.

Live, uncompressed music often has peaks that are 4 dB higher, or
more, than a typical commercial CD. Such peaks are brief, and would
not really affect the total size of the FLAC file. Really, the
average level is what determines whether the FLAC file ends up being
smaller than 50% as it is in this case.

Post by scott brown
He sent me the original wav this morning and I loaded it into Wave
Editor on OS X. I dithered to 16 bit using MBIT+ (high/ultra
setting) and saved the 16 bit file. I did nothing else (no
normalizing or any other processing). I can't give you the 16 bit
size right now since I'm at work and the file is on my Mac at home,
but I can report back tonight.

Impressive! I have read many comments that MBIT+ is the best. I've
only recently licensed it myself, so I have not yet had time to form
a personal opinion as to whether it is better than the dither that I
have been using for years.

No matter how good the dither is, though, it's still noise. The
human ear and brain system cannot hear MBIT+, but FLAC is just a
mathematical process. Dithering from 24-bit to 16-bit is equivalent
to increasing the quantization noise by about 48 dB! It's actually
quite impressive that you can add 48 dB of noise and the FLAC file
only increases in size by less than 2%.

Thanks for the details. I'm curious about the file size, but
uncompressed WAV should be exactly as I predicted.

Post by scott brown
Whatever my process is, though, the guy who originally recorded the
file gets the same results with whatever method he uses to convert
to 16 bit on Windows. I can ask him what his 24 > 16 bit process
is. I can also just truncate the file down to 16 bit and report
back on the resulting flac file size. Would you expect that flac
file to be around the same size as the 24 bit? In my experience,
my 24/48 flac files are always substantialy bigger than my 16 bit
flac files, which is why this case confuses me...

Personally, I rarely pay close attention to the exact compression.
I'm happy just that FLAC is smaller and lossless. But I am still
curious about the various reasons why some files turn out bigger or
smaller than others. I tend to do everything in 24-bit, even final
mastering, so I have not looked at 16-bit in a long while.

I can say that DTS surround music disc, which is 14-bit data in 16-
bit CD format, does end up with a FLAC that is almost exactly 87.5%
of the WAV. This makes perfect sense, because the DTS data looks
like random white noise to FLAC, and the only thing FLAC can do is
compress those 2 unused bits.

It's tempting to look at the 24-bit to 16-bit conversion as simply
dropping 1/3 of the data, since files are based on 8-bit bytes. But
one way to look at this is that FLAC deals with audio samples as if
they were all 32-bit. The 16-bit samples simply have more noise. It
will be interesting to see what happens with your 16-bit truncation
test. I would expect that the FLAC would only get smaller, not
increase in size, if all you do is truncate.

Fortunately, FLAC is quite smart about bit utilization, and can even
detect 16-bit samples in a 24-bit file. I don't think that's
happening with your files, though, because I would expect the 24-bit
FLAC to be around 33% of the 24-bit WAV if it actually only had 16-
bit samples, instead of the 40% that you're seeing. But I suppose
that's always possible.

Brian Willoughby
Sound Consulting

P.S. I'm still curious where Nicholas came up with the 14% value.
Is that based on decibels, bits, or some other metric I haven't
thought of?

[scott brown]

original 24/48 wav file: 264,904,968 bytes
flac level 8: 105,992,780 bytes

dithered 16/48 wav file:173,885,996 bytes
flac level 8: 108,700,948 bytes

truncated 16/48 wav file: 173,885,996 bytes
flac level 8: 105,224,448 bytes

RMS level of original 24 bit: -15.3dB with peaks at -0.3dB

if I normalize the original file to a max of 0.0, the resulting flac file is
192,798,482 bytes. weird....

Scott

Post by scott brown
My first thought was that the file had low levels (before he sent me the
file), but that's definitely not the case with this file. There are many
peaks that reach 0dBFS.

Live, uncompressed music often has peaks that are 4 dB higher, or more,
than a typical commercial CD. Such peaks are brief, and would not really
affect the total size of the FLAC file. Really, the average level is what
determines whether the FLAC file ends up being smaller than 50% as it is in
this case.
He sent me the original wav this morning and I loaded it into Wave Editor

Post by scott brown
on OS X. I dithered to 16 bit using MBIT+ (high/ultra setting) and saved
the 16 bit file. I did nothing else (no normalizing or any other
processing). I can't give you the 16 bit size right now since I'm at work
and the file is on my Mac at home, but I can report back tonight.

Impressive! I have read many comments that MBIT+ is the best. I've only
recently licensed it myself, so I have not yet had time to form a personal
opinion as to whether it is better than the dither that I have been using
for years.
No matter how good the dither is, though, it's still noise. The human ear
and brain system cannot hear MBIT+, but FLAC is just a mathematical process.
Dithering from 24-bit to 16-bit is equivalent to increasing the
quantization noise by about 48 dB! It's actually quite impressive that you
can add 48 dB of noise and the FLAC file only increases in size by less than
2%.
Thanks for the details. I'm curious about the file size, but uncompressed
WAV should be exactly as I predicted.
Whatever my process is, though, the guy who originally recorded the file

Post by scott brown
gets the same results with whatever method he uses to convert to 16 bit on
Windows. I can ask him what his 24 > 16 bit process is. I can also just
truncate the file down to 16 bit and report back on the resulting flac file
size. Would you expect that flac file to be around the same size as the 24
bit? In my experience, my 24/48 flac files are always substantialy bigger
than my 16 bit flac files, which is why this case confuses me...

Personally, I rarely pay close attention to the exact compression. I'm
happy just that FLAC is smaller and lossless. But I am still curious about
the various reasons why some files turn out bigger or smaller than others.
I tend to do everything in 24-bit, even final mastering, so I have not
looked at 16-bit in a long while.
I can say that DTS surround music disc, which is 14-bit data in 16-bit CD
format, does end up with a FLAC that is almost exactly 87.5% of the WAV.
This makes perfect sense, because the DTS data looks like random white
noise to FLAC, and the only thing FLAC can do is compress those 2 unused
bits.
It's tempting to look at the 24-bit to 16-bit conversion as simply dropping
1/3 of the data, since files are based on 8-bit bytes. But one way to look
at this is that FLAC deals with audio samples as if they were all 32-bit.
The 16-bit samples simply have more noise. It will be interesting to see
what happens with your 16-bit truncation test. I would expect that the FLAC
would only get smaller, not increase in size, if all you do is truncate.
Fortunately, FLAC is quite smart about bit utilization, and can even detect
16-bit samples in a 24-bit file. I don't think that's happening with your
files, though, because I would expect the 24-bit FLAC to be around 33% of
the 24-bit WAV if it actually only had 16-bit samples, instead of the 40%
that you're seeing. But I suppose that's always possible.
Brian Willoughby
Sound Consulting
P.S. I'm still curious where Nicholas came up with the 14% value. Is that
based on decibels, bits, or some other metric I haven't thought of?

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.xiph.org/pipermail/flac/attachments/20101202/07207870/attachment.htm

[Brian Willoughby]

Post by scott brown
original 24/48 wav file: 264,904,968 bytes
flac level 8: 105,992,780 bytes
dithered 16/48 wav file:173,885,996 bytes
flac level 8: 108,700,948 bytes
truncated 16/48 wav file: 173,885,996 bytes
flac level 8: 105,224,448 bytes
RMS level of original 24 bit: -15.3dB with peaks at -0.3dB
if I normalize the original file to a max of 0.0, the resulting
flac file is 192,798,482 bytes. weird....

Normalizing will probably always make the FLAC larger. Of the many
components of the FLAC algorithm, one is to use differential values
rather than absolute, and another is to use variable-length coding
(Rice coding). Quieter files have smaller differential sample
values, and thus compress more.

I am surprised that a mere 0.3 dB normalization would add over 80% to
the FLAC size. Perhaps you missed something? Can the RMS level
actually reach -0.3 dB without serious distortion? ... or did you
mean -15.3 dB RMS and -0.3 dB PPM?

Comparing the 24-bit WAV to the 16-bit WAV, it looks like as much as
4 MB of non-audio data is in the file. You may be looking at
waveform overviews or other extra chunks in the WAV file which are
usually discarded by FLAC. But this only explains a small part of
the surprising numbers.

One thing that stands out to me is that your original 24/48 WAV may
not actually have 24-bit samples in it. I wrote a program which
could detect this situation (16-bit samples in a 24-bit file), but I
do not know of any other available tool to test this. It seems very
suspicious that the 24-bit FLAC and truncated 16-bit FLAC are within
0.73% of each other (i.e., less than 1%). I have a suspicion that
your friend is recording 24-bit files from a 16-bit A/D converter
interface, or at least the software is set for 24-bit files while the
interface is set to 16-bit mode. You should have the recordist
double-check all settings. A mistake here would easily explain the
strange FLAC compression ratios.

Brian Willoughby
Sound Consulting

[scott brown]

I meant -15.3 RMS and -.3 peak.

And bingo, you were correct in your last guess. I feel dumb for not
thinking of it myself. He's using a 24 bit interface, but he must not be
outputting 24 bits because when I tested the file, it's only actually 16
bits of audio. Next time, that's the first thing I'm checking :)

Thanks again for all the help. I knew something about this didn't seem
right...

Scott

Post by Brian Willoughby

Post by scott brown
original 24/48 wav file: 264,904,968 bytes
flac level 8: 105,992,780 bytes
dithered 16/48 wav file:173,885,996 bytes
flac level 8: 108,700,948 bytes
truncated 16/48 wav file: 173,885,996 bytes
flac level 8: 105,224,448 bytes
RMS level of original 24 bit: -15.3dB with peaks at -0.3dB
if I normalize the original file to a max of 0.0, the resulting flac file
is 192,798,482 bytes. weird....

Normalizing will probably always make the FLAC larger. Of the many
components of the FLAC algorithm, one is to use differential values rather
than absolute, and another is to use variable-length coding (Rice coding).
Quieter files have smaller differential sample values, and thus compress
more.
I am surprised that a mere 0.3 dB normalization would add over 80% to the
FLAC size. Perhaps you missed something? Can the RMS level actually reach
-0.3 dB without serious distortion? ... or did you mean -15.3 dB RMS and
-0.3 dB PPM?
Comparing the 24-bit WAV to the 16-bit WAV, it looks like as much as 4 MB
of non-audio data is in the file. You may be looking at waveform overviews
or other extra chunks in the WAV file which are usually discarded by FLAC.
But this only explains a small part of the surprising numbers.
One thing that stands out to me is that your original 24/48 WAV may not
actually have 24-bit samples in it. I wrote a program which could detect
this situation (16-bit samples in a 24-bit file), but I do not know of any
other available tool to test this. It seems very suspicious that the 24-bit
FLAC and truncated 16-bit FLAC are within 0.73% of each other (i.e., less
than 1%). I have a suspicion that your friend is recording 24-bit files
from a 16-bit A/D converter interface, or at least the software is set for
24-bit files while the interface is set to 16-bit mode. You should have the
recordist double-check all settings. A mistake here would easily explain
the strange FLAC compression ratios.
Brian Willoughby
Sound Consulting

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.xiph.org/pipermail/flac/attachments/20101202/2afa9bc1/attachment-0001.htm