[Flac] Fixing corrupt flac files

[Knut Krause]

Hi,

since years I simply encoded my CDs using various tools to flac and simply
listened to it and never had any problems.

Today I wrote a little script though to add replaygain values to my music
collection and I found out that a lot of my flac files are corrupt.

The corruption almost always occurs in the last 10% like

flac 1.2.1, Copyright (C) 2000,2001,2002,2003,2004,2005,2006,2007 Josh Coalson
flac comes with ABSOLUTELY NO WARRANTY. This is free software, and you are
welcome to redistribute it under certain conditions. Type `flac' for details.

01 Overture.flac: testing, 96% complete

01 Overture.flac: ERROR while decoding data
state = FLAC__STREAM_DECODER_END_OF_STREAM

Is there a way to fix this? I tested decoding the file with -F and then
reencoding it with flac. This seems to lead to an error free version but is
this the way to go?

Regards


Knut

[Martijn van Beurden]

Hi,

Do you still know what tools you used to encode these FLAC-files? It
seems similar errors have popped up elsewhere after using various
tagging-utilities. It seems it is indeed a tagging issue, as the error
indicates something in that direction and the errors always occur at the
last few percent. I guess it isn't possible to share one of these files
somewhere so I can have a look?

If you are sure the files are actually unharmed and you want to
re-encode them, you can use flac -f -F file.flac. This will re-encode
the files in place, overwriting the original, so be sure to keep a copy
of the original until you are sure this is indeed the way to go. The big
advantage of this method is that you keep all tags, if the file has any.

Post by Knut Krause
Hi,
since years I simply encoded my CDs using various tools to flac and simply
listened to it and never had any problems.
Today I wrote a little script though to add replaygain values to my music
collection and I found out that a lot of my flac files are corrupt.
The corruption almost always occurs in the last 10% like
flac 1.2.1, Copyright (C) 2000,2001,2002,2003,2004,2005,2006,2007 Josh Coalson
flac comes with ABSOLUTELY NO WARRANTY. This is free software, and you are
welcome to redistribute it under certain conditions. Type `flac' for details.
01 Overture.flac: testing, 96% complete
01 Overture.flac: ERROR while decoding data
state = FLAC__STREAM_DECODER_END_OF_STREAM
Is there a way to fix this? I tested decoding the file with -F and then
reencoding it with flac. This seems to lead to an error free version but is
this the way to go?
Regards
Knut
_______________________________________________
Flac mailing list
Flac at xiph.org
http://lists.xiph.org/mailman/listinfo/flac

[Knut Krause]

Hej,

well I'm not sure because those file that are corrupt have been around for a
quite long time. It could have been k3b but I'm really not sure. Right now I
was ripping with Audex and tagging with Audex/Amarok/kid3. Those files seem to
be fine though.

My lates approaches have been with Picard/beets :-)

I think it's not possible to share those files without getting in legal trouble
here or is it possible somehow to extract some information so it won't be an
issue?

regards


Knut

Post by Martijn van Beurden
Hi,
Do you still know what tools you used to encode these FLAC-files? It
seems similar errors have popped up elsewhere after using various
tagging-utilities. It seems it is indeed a tagging issue, as the error
indicates something in that direction and the errors always occur at the
last few percent. I guess it isn't possible to share one of these files
somewhere so I can have a look?
If you are sure the files are actually unharmed and you want to
re-encode them, you can use flac -f -F file.flac. This will re-encode
the files in place, overwriting the original, so be sure to keep a copy
of the original until you are sure this is indeed the way to go. The big
advantage of this method is that you keep all tags, if the file has any.

Post by Knut Krause
Hi,
since years I simply encoded my CDs using various tools to flac and simply
listened to it and never had any problems.
Today I wrote a little script though to add replaygain values to my music
collection and I found out that a lot of my flac files are corrupt.
The corruption almost always occurs in the last 10% like
flac 1.2.1, Copyright (C) 2000,2001,2002,2003,2004,2005,2006,2007 Josh
Coalson flac comes with ABSOLUTELY NO WARRANTY. This is free software,
and you are welcome to redistribute it under certain conditions. Type
`flac' for details.
01 Overture.flac: testing, 96% complete
01 Overture.flac: ERROR while decoding data
state = FLAC__STREAM_DECODER_END_OF_STREAM
Is there a way to fix this? I tested decoding the file with -F and then
reencoding it with flac. This seems to lead to an error free version but is
this the way to go?
Regards
Knut
_______________________________________________
Flac mailing list
Flac at xiph.org
http://lists.xiph.org/mailman/listinfo/flac

_______________________________________________
Flac mailing list
Flac at xiph.org
http://lists.xiph.org/mailman/listinfo/flac

[Martijn van Beurden]

Post by Knut Krause
I think it's not possible to share those files without getting in legal trouble
here or is it possible somehow to extract some information so it won't be an
issue?

It could have been you had a file with the problem which is released
under a Creative Commons licence, but that's not very common. Just asking.

You can do a few things. Reading your previous post, it seems you now
how to work with a command line, so could you post the output of the
following two commands:

flac -a -F some_corrupt_file.flac (see the .ana file which is generated)
metaflac --list some_corrupt_file.flac (just the console output)

Plus the exact size (in bytes) of the FLAC-file and the WAV-file that
you retrieve using flac -d -F some_corrupt_file.flac

You can also try to cut off the last 500kb of the FLAC-file if you are
using Linux (or other UNIX-like probably, I have no idea how to do this
in Windows or Mac)
tail -c 500000 some_corrupt_file.flac > tail_cut_off.flac
That's a sample of at most 10 seconds and won't get you into trouble.

[Knut Krause]

So, let's provide some information then :-)

----------------------------------------------------------------------------------------------

soa2ii at thor /mnt/files/music/Slime/Alle gegen Alle $ flac -aF 02\
St?rtebecker.flac

flac 1.2.1, Copyright (C) 2000,2001,2002,2003,2004,2005,2006,2007 Josh Coalson
flac comes with ABSOLUTELY NO WARRANTY. This is free software, and you are
welcome to redistribute it under certain conditions. Type `flac' for details.

02 St?rtebecker.flac: analyzing, 96% complete

02 St?rtebecker.flac: ERROR while decoding data
state = FLAC__STREAM_DECODER_END_OF_STREAM
02 St?rtebecker.flac: ERROR, MD5 signature mismatch

----------------------------------------------------------------------------------------------

The *.ana: http://bpaste.net/show/69346/

----------------------------------------------------------------------------------------------

The size in bytes:
soa2ii at thor /mnt/files/music/Slime/Alle gegen Alle $ du 02\ St?rtebecker.flac
16680 02 St?rtebecker.flac

----------------------------------------------------------------------------------------------

The size of the wav file
soa2ii at thor /mnt/files/music/Slime/Alle gegen Alle $ flac -dF 02\
St?rtebecker.flac

flac 1.2.1, Copyright (C) 2000,2001,2002,2003,2004,2005,2006,2007 Josh Coalson
flac comes with ABSOLUTELY NO WARRANTY. This is free software, and you are
welcome to redistribute it under certain conditions. Type `flac' for details.

02 St?rtebecker.flac: 96% complete

02 St?rtebecker.flac: ERROR while decoding data
state = FLAC__STREAM_DECODER_END_OF_STREAM
02 St?rtebecker.flac: ERROR, MD5 signature mismatch
soa2ii at thor /mnt/files/music/Slime/Alle gegen Alle $ du 02\ St?rtebecker.wav
22468 02 St?rtebecker.wav

------------------------------------------------------------------------------------------------

The tail of the corrupted file: http://lagom.de/tmp/tail.flac

I hope this is all information needed :-)

regards


Knut

Post by Martijn van Beurden

Post by Knut Krause
I think it's not possible to share those files without getting in legal
trouble here or is it possible somehow to extract some information so it
won't be an issue?

It could have been you had a file with the problem which is released
under a Creative Commons licence, but that's not very common. Just asking.
You can do a few things. Reading your previous post, it seems you now
how to work with a command line, so could you post the output of the
flac -a -F some_corrupt_file.flac (see the .ana file which is generated)
metaflac --list some_corrupt_file.flac (just the console output)
Plus the exact size (in bytes) of the FLAC-file and the WAV-file that
you retrieve using flac -d -F some_corrupt_file.flac
You can also try to cut off the last 500kb of the FLAC-file if you are
using Linux (or other UNIX-like probably, I have no idea how to do this
in Windows or Mac)
tail -c 500000 some_corrupt_file.flac > tail_cut_off.flac
That's a sample of at most 10 seconds and won't get you into trouble.
_______________________________________________
Flac mailing list
Flac at xiph.org
http://lists.xiph.org/mailman/listinfo/flac

[Martijn van Beurden]

Post by Knut Krause
02 St?rtebecker.flac: ERROR while decoding data
state = FLAC__STREAM_DECODER_END_OF_STREAM
02 St?rtebecker.flac: ERROR, MD5 signature mismatch

Ah, you didn't mention that last one, MD5 mismatch, in your first mail.

Post by Knut Krause
The tail of the corrupted file: http://lagom.de/tmp/tail.flac

I haven't been able to pinpoint the cause of this error, (I'm able to
reproduce it) but it certainly is no tagging issue, which I thought at
first, so a corruption seems the cause. However, corruption happening
with a lot of files, all in the last few percent, that cannot be
coincidence. I'm definitely going to look into this, probably tomorrow
or somewhere this weekend, as trying to decode the file "by hand" to
find the error takes a lot of time.

Could you please send me the output of metaflac --list as well, in
particular the vendor string from the VORBIS_COMMENT block, I'm curious
which encoder produced this.

[Knut Krause]

Here you are:

soa2ii at thor /mnt/files/music/Slime/Alle gegen Alle $ metaflac --list 02\
St?rtebecker.flac
METADATA block #0
type: 0 (STREAMINFO)
is last: false
length: 34
minimum blocksize: 4608 samples
maximum blocksize: 4608 samples
minimum framesize: 14 bytes
maximum framesize: 15637 bytes
sample_rate: 44100 Hz
channels: 2
bits-per-sample: 16
total samples: 5857656
MD5 signature: eed0112053a62bbe0734502ef9a4b0b0
METADATA block #1
type: 3 (SEEKTABLE)
is last: false
length: 234
seek points: 13
point 0: sample_number=0, stream_offset=0, frame_samples=4608
point 1: sample_number=446976, stream_offset=1314129, frame_samples=4608
point 2: sample_number=898560, stream_offset=2663921, frame_samples=4608
point 3: sample_number=1350144, stream_offset=4059085, frame_samples=4608
point 4: sample_number=1801728, stream_offset=5431315, frame_samples=4608
point 5: sample_number=2248704, stream_offset=6720588, frame_samples=4608
point 6: sample_number=2700288, stream_offset=8088365, frame_samples=4608
point 7: sample_number=3151872, stream_offset=9456245, frame_samples=4608
point 8: sample_number=3603456, stream_offset=10823115, frame_samples=4608
point 9: sample_number=4055040, stream_offset=12179598, frame_samples=4608
point 10: sample_number=4502016, stream_offset=13497513, frame_samples=4608
point 11: sample_number=4953600, stream_offset=14827454, frame_samples=4608
point 12: sample_number=5405184, stream_offset=16137500, frame_samples=4608
METADATA block #2
type: 4 (VORBIS_COMMENT)
is last: false
length: 701
vendor string: reference libFLAC 1.1.2 20050205
comments: 25
comment[0]: title=St?rtebecker
comment[1]: releasecountry=DE
comment[2]: totaldiscs=1
comment[3]: label=Aggressive Rockproduktionen
comment[4]: totaltracks=15
comment[5]: musicbrainz_albumartistid=a0fe06e3-deb7-4ea2-a7de-cc93740fd163
comment[6]: date=1983
comment[7]: discnumber=1
comment[8]: tracktotal=15
comment[9]: asin=B00000AQ1Q
comment[10]: albumartistsort=Slime
comment[11]: originaldate=1983
comment[12]: artist=Slime
comment[13]: musicbrainz_albumid=66babc5d-cd05-41ec-b96f-9f48ce56a81f
comment[14]: releasestatus=official
comment[15]: albumartist=Slime
comment[16]: album=Alle gegen Alle
comment[17]: musicbrainz_artistid=a0fe06e3-deb7-4ea2-a7de-cc93740fd163
comment[18]: media=CD
comment[19]: releasetype=album
comment[20]: disctotal=1
comment[21]: script=Latn
comment[22]: musicbrainz_trackid=123617d2-5c5f-4926-91ba-59b9ebcc90fe
comment[23]: artistsort=Slime
comment[24]: tracknumber=2
METADATA block #3
type: 1 (PADDING)
is last: true
length: 3559


I'll provide flac -t of the whole album. It looks to me as if there are a lot
of error types present ;-)

soa2ii at thor /mnt/files/music/Slime/Alle gegen Alle $ flac -t *.flac

flac 1.2.1, Copyright (C) 2000,2001,2002,2003,2004,2005,2006,2007 Josh Coalson
flac comes with ABSOLUTELY NO WARRANTY. This is free software, and you are
welcome to redistribute it under certain conditions. Type `flac' for details.

01 Linke Spie?er.flac: testing, 94% complete

01 Linke Spie?er.flac: ERROR while decoding data
state = FLAC__STREAM_DECODER_END_OF_STREAM
02 St?rtebecker.flac: testing, 96% complete

02 St?rtebecker.flac: ERROR while decoding data
state = FLAC__STREAM_DECODER_END_OF_STREAM
03 Untergang.flac: testing, 97% complete

03 Untergang.flac: ERROR while decoding data
state = FLAC__STREAM_DECODER_END_OF_STREAM
04 Zu kalt.flac: testing, 98% complete

04 Zu kalt.flac: ERROR while decoding data
state = FLAC__STREAM_DECODER_END_OF_STREAM
05 Ihr seid sch?n.flac: ERROR, MD5 signature mismatch
06 Religion.flac: testing, 99% complete

06 Religion.flac: ERROR while decoding data
state = FLAC__STREAM_DECODER_END_OF_STREAM
07 Nazis raus.flac: testing, 96% complete07 Nazis raus.flac: *** Got error code
1:FLAC__STREAM_DECODER_ERROR_STATUS_BAD_HEADER
07 Nazis raus.flac: *** Got error code
0:FLAC__STREAM_DECODER_ERROR_STATUS_LOST_SYNC


07 Nazis raus.flac: ERROR while decoding data
state = FLAC__STREAM_DECODER_READ_FRAME
08 Sand im Getriebe.flac: testing, 98% complete

08 Sand im Getriebe.flac: ERROR while decoding data
state = FLAC__STREAM_DECODER_END_OF_STREAM
09 Alle gegen Alle.flac: testing, 96% complete09 Alle gegen Alle.flac: *** Got
error code 0:FLAC__STREAM_DECODER_ERROR_STATUS_LOST_SYNC
09 Alle gegen Alle.flac: *** Got error code
0:FLAC__STREAM_DECODER_ERROR_STATUS_LOST_SYNC


09 Alle gegen Alle.flac: ERROR while decoding data
state = FLAC__STREAM_DECODER_READ_FRAME
10 Die Letzten.flac: testing, 98% complete10 Die Letzten.flac: *** Got error
code 2:FLAC__STREAM_DECODER_ERROR_STATUS_FRAME_CRC_MISMATCH


10 Die Letzten.flac: ERROR while decoding data
state = FLAC__STREAM_DECODER_READ_FRAME
11 Etikette t?tet.flac: testing, 97% complete11 Etikette t?tet.flac: *** Got
error code 0:FLAC__STREAM_DECODER_ERROR_STATUS_LOST_SYNC
11 Etikette t?tet.flac: *** Got error code
0:FLAC__STREAM_DECODER_ERROR_STATUS_LOST_SYNC


11 Etikette t?tet.flac: ERROR while decoding data
state = FLAC__STREAM_DECODER_READ_FRAME
12 Ich will nicht werden.flac: testing, 99% complete

12 Ich will nicht werden.flac: ERROR while decoding data
state = FLAC__STREAM_DECODER_END_OF_STREAM
13 Tod.flac: testing, 96% complete

13 Tod.flac: ERROR while decoding data
state = FLAC__STREAM_DECODER_END_OF_STREAM
14 Junge Junge.flac: testing, 92% complete

14 Junge Junge.flac: ERROR while decoding data
state = FLAC__STREAM_DECODER_END_OF_STREAM
15 Guter Rat ist teuer.flac: testing, 97% complete

15 Guter Rat ist teuer.flac: ERROR while decoding data
state = FLAC__STREAM_DECODER_END_OF_STREAM


I wrote a little script checking my whole collection for corrupt files using

flac -wst "$1" 2>/dev/null || printf '%3d %s\n' "$?" "$1"

in a little find/bash script. I can run batch checks and provide output as
needed :-)

regards


Knut

Post by Martijn van Beurden

Post by Knut Krause
02 St?rtebecker.flac: ERROR while decoding data
state = FLAC__STREAM_DECODER_END_OF_STREAM
02 St?rtebecker.flac: ERROR, MD5 signature mismatch

Ah, you didn't mention that last one, MD5 mismatch, in your first mail.

Post by Knut Krause
The tail of the corrupted file: http://lagom.de/tmp/tail.flac

I haven't been able to pinpoint the cause of this error, (I'm able to
reproduce it) but it certainly is no tagging issue, which I thought at
first, so a corruption seems the cause. However, corruption happening
with a lot of files, all in the last few percent, that cannot be
coincidence. I'm definitely going to look into this, probably tomorrow
or somewhere this weekend, as trying to decode the file "by hand" to
find the error takes a lot of time.
Could you please send me the output of metaflac --list as well, in
particular the vendor string from the VORBIS_COMMENT block, I'm curious
which encoder produced this.
_______________________________________________
Flac mailing list
Flac at xiph.org
http://lists.xiph.org/mailman/listinfo/flac

[Martijn van Beurden]

Post by Knut Krause
[...]
01 Linke Spie?er.flac: testing, 94% complete
01 Linke Spie?er.flac: ERROR while decoding data
state = FLAC__STREAM_DECODER_END_OF_STREAM
[...]
05 Ihr seid sch?n.flac: ERROR, MD5 signature mismatch
[...]
07 Nazis raus.flac: testing, 96% complete07 Nazis raus.flac: *** Got error code
1:FLAC__STREAM_DECODER_ERROR_STATUS_BAD_HEADER
07 Nazis raus.flac: *** Got error code
0:FLAC__STREAM_DECODER_ERROR_STATUS_LOST_SYNC
07 Nazis raus.flac: ERROR while decoding data
state = FLAC__STREAM_DECODER_READ_FRAME
[...]
10 Die Letzten.flac: testing, 98% complete10 Die Letzten.flac: *** Got error
code 2:FLAC__STREAM_DECODER_ERROR_STATUS_FRAME_CRC_MISMATCH
10 Die Letzten.flac: ERROR while decoding data
state = FLAC__STREAM_DECODER_READ_FRAME
11 Etikette t?tet.flac: testing, 97% complete11 Etikette t?tet.flac: *** Got
error code 0:FLAC__STREAM_DECODER_ERROR_STATUS_LOST_SYNC
11 Etikette t?tet.flac: *** Got error code
0:FLAC__STREAM_DECODER_ERROR_STATUS_LOST_SYNC
11 Etikette t?tet.flac: ERROR while decoding data
state = FLAC__STREAM_DECODER_READ_FRAME

This is really weird: all kinds of different errors, but indeed all in
the last few percent. The small part you send looks fine: there's
probably a corruption somewhere, but it's not obvious. The strange thing
is that the frame it happened is actually the frame of the fade-out: all
previous frames contain audio, the frames that follow it are all zero.

There is no (feasible) way to fix this, unfortunately. If tried to find
the cause of the problem, but I can't be conclusive. First: in the

Post by Knut Krause
Fixed a bug where sync error at end-of-stream of truncated files was
not being caught

It might be that your ripper produced not properly terminated files
which 'silently chrashed' FLAC. It might be k3b (if you indeed used that
program as for ripping) as it has had some issues with FLAC encoding and
decoding in the past (I've had some of those problems myself) see here:
https://bugs.kde.org/buglist.cgi?bug_status=__all__&product=k3b&content=flac

I think this is the only possible explanation: this can't be failing
storage media or file transfer methods, it's just too regular, so this
problem has to have occurred at encoding them.

Post by Knut Krause
I wrote a little script checking my whole collection for corrupt files using
flac -wst "$1" 2>/dev/null || printf '%3d %s\n' "$?" "$1"
in a little find/bash script. I can run batch checks and provide output as
needed :-)

Did you find more corrupt files?

Anyway, I'll send this case to the developers, they might find a cause,
but I think your files can't be fixed, you'll have to re-rip them or
re-encode them without the last tenths of a second of the file.

[Linda Walsh]

Post by Knut Krause
So, let's provide some information then :-)
----------------------------------------------------------------------------------------------
soa2ii at thor /mnt/files/music/Slime/Alle gegen Alle $ flac -aF 02\
St?rtebecker.flac

Could this have something to do with you encoding the song titles
in ISO-8859-1 and flac-meta information is in UTF-8?

[Knut Krause]

Here everything looks fine. Could it be a local encoding problem on your side?

Post by Linda Walsh

Post by Knut Krause
So, let's provide some information then :-)
--------------------------------------------------------------------------
--------------------
soa2ii at thor /mnt/files/music/Slime/Alle gegen Alle $ flac -aF 02\
St?rtebecker.flac

Could this have something to do with you encoding the song titles
in ISO-8859-1 and flac-meta information is in UTF-8?