rsync not syncing some files

[lewashby]

I've been using rsync and or luckybackup for a while now to sync my desktop's music library to my micro sd card for my digital music player. Today I noticed that my music player wasn't showing one of my albums so I've started investigating and I've found that it's not the music player but is more likely to be rsync, but I don't understand how or why. After syncing I check the sd card manually and sure enough this album is still not beeing synced.

On my computer all of my music is stored at /home/me/media/music/* inside of this last music directory there are two various artists directories, one is uppercase and one is lower case for the word artists. Variouse Artists & Variouse artists. The weird thing is my sd card is only getting the contents of the 'Various artists' directory that's on my desktop but on my sd card the directory is called 'Various Artists' and there is not 'Various artists' directory listed at all on my sd card. So I'm only getting one of the two copies on my sync and it's filled with the contents that the other one is filled with on my computer. In this case I used luckbackup but it uses rysnc, I'll past the code below. Any ideas? Thanks. Oh, and sd card is fat32.

rsync -h --progress --stats -r -t --modify-window=1 --delete-after --delete-excluded --exclude=**/lost+found*/ --exclude=**/*Trash*/ --exclude=**/*trash*/ /home/me/media/music/ /media/me/MEDIA-SD/

[rproffitt]

It's a long shot but how is the card mounted?

Read https://stackoverflow.com/questions/28398905/rsync-copies-unchanged-files-usb-drive-fat32 about the 'shortname=mixed' option.

[rubberman]

Also, make sure the directory and associated files are readable. If they aren't readable, then you can have this problem. I have had similar issues in the past. Make sure the directory also has full rwx permissions.

[lewashby]

Yes it's mounted, otherwise I wouldn't be able to navigate to the SD card and list it's contents and luckBackup wouldn't even let me run the sync if it's wasn't. And everything should be rwx but just to me sure I just ran chmod -R +rwx ~/media/music/*. Check this out, after running that command I seen the following and the end of my luckybackup log.

         13.01M 100%   15.77MB/s    0:00:00 (xfr#1479, to-chk=210/6213)
Various artists/

deleting .audio_data/album_img/52407A71DB71DB0606836602A75A619E
deleting .luckybackup-snaphots/default.profile/snaps/default-music-sync2-20170530202113.changes.log
deleting .luckybackup-snaphots/default.profile/snaps/
deleting .luckybackup-snaphots/default.profile/profiles/default.profile
deleting .luckybackup-snaphots/default.profile/profiles/
deleting .luckybackup-snaphots/default.profile/logs/default-music-sync2-20170530202113.log
deleting .luckybackup-snaphots/default.profile/logs/
deleting .luckybackup-snaphots/default.profile/
deleting Coldplay/A Rush of Blood to the Head/11. Amsterdam.flac
deleting Coldplay/A Rush of Blood to the Head/10. A Rush Of Blood To The Head.flac
deleting Coldplay/A Rush of Blood to the Head/09. A Whisper.flac
deleting Coldplay/A Rush of Blood to the Head/08. Warning Sign.flac
deleting Coldplay/A Rush of Blood to the Head/07. Green Eyes.flac
deleting Coldplay/A Rush of Blood to the Head/06. Daylight.flac
deleting Coldplay/A Rush of Blood to the Head/05. Clocks.flac
deleting Coldplay/A Rush of Blood to the Head/04. The Scientist.flac
deleting Coldplay/A Rush of Blood to the Head/03. God Put A Smile Upon Your Face.flac
deleting Coldplay/A Rush of Blood to the Head/02. In My Place.flac
deleting Coldplay/A Rush of Blood to the Head/01. Politik.flac
deleting Various artists/White Christmas/10. Deck the Halls.flac
deleting Various artists/White Christmas/09. O Little Town of Bethlehem.flac
deleting Various artists/White Christmas/08. It's Beginning to Look a Lot Like Christmas.flac
deleting Various artists/White Christmas/07. Winter Wonderland.flac
deleting Various artists/White Christmas/06. Frosty the Snowman.flac
deleting Various artists/White Christmas/05. A Holly Jolly Christmas.flac
deleting Various artists/White Christmas/04. Toyland.flac
deleting Various artists/White Christmas/03. Sleigh Ride.flac
deleting Various artists/White Christmas/02. Jingle Bells.flac
deleting Various artists/White Christmas/01. White Christmas.flac
deleting Various artists/White Christmas/
deleting Various artists/The Walking Dead_ AMC Original Soundtrack, Vol. 2/04. Be Not So Fearful.mp3
deleting Various artists/The Walking Dead_ AMC Original Soundtrack, Vol. 2/
deleting Various artists/Sentimentally Scottish/Scottish Soldier.mp3
deleting Various artists/Sentimentally Scottish/
deleting Various artists/Rain Man _ Original Motion Picture Soundtrack/01. Iko Iko.mp3
deleting Various artists/Rain Man _ Original Motion Picture Soundtrack/
deleting Various artists/Rain Man _ Original Motion Picture Sound/01 Iko Iko.mp3
deleting Various artists/Rain Man _ Original Motion Picture Sound/
deleting Various artists/Pop a Paris - Rock N' Roll and Mini Skirts Vol.1/03. Ces bottes sont faites pour marcher.mp3
deleting Various artists/Pop a Paris - Rock N' Roll and Mini Skirts Vol.1/
deleting Various artists/Platinum Christmas/07 Little Drummer Boy.mp3
deleting Various artists/Platinum Christmas/
deleting Various artists/One Hit Wonders/04. Rockin' Little Angel.mp3
deleting Various artists/One Hit Wonders/
deleting Various artists/O Brother, Where Art Thou_/10. Didn't Leave Nobody But The Baby (O Brother, Where Art Thou_ Soundtrack).mp3
deleting Various artists/O Brother, Where Art Thou_/05. I Am A Man Of Constant Sorrow (O Brother, Where Art Thou_ Soundtrack_Radio Station Version) [feat. Dan Tyminski].mp3
deleting Various artists/O Brother, Where Art Thou_/04. Down To The River To Pray (O Brother, Where Art Thou_ Soundtrack).mp3
deleting Various artists/O Brother, Where Art Thou_/02. Big Rock Candy Mountain (O Brother, Where Art Thou_ Soundtrack).mp3
deleting Various artists/O Brother, Where Art Thou_/
deleting Various artists/Now That's What I Call Music! 15/17. Feeling This.flac
deleting Various artists/Now That's What I Call Music! 15/
deleting Various artists/Now That's What I Call Music 15/20. 100 Years.flac
deleting Various artists/Now That's What I Call Music 15/19. Falls on Me.flac
deleting Various artists/Now That's What I Call Music 15/18. Hold On.flac

deleting Various artists/Now That's What I Call Music 15/16. (I Hate) Everything About You.flac
deleting Various artists/Now That's What I Call Music 15/15. Everything.flac
deleting Various artists/Now That's What I Call Music 15/14. The First Cut Is the Deepest.flac
deleting Various artists/Now That's What I Call Music 15/13. Sunrise.flac
deleting Various artists/Now That's What I Call Music 15/12. With You.flac
deleting Various artists/Now That's What I Call Music 15/11. I Don't Want You Back.flac
deleting Various artists/Now That's What I Call Music 15/10. Shorty Doowop.flac
deleting Various artists/Now That's What I Call Music 15/09. Me, Myself and I.flac
deleting Various artists/Now That's What I Call Music 15/08. Gigolo.flac
deleting Various artists/Now That's What I Call Music 15/07. Shut Up.flac
deleting Various artists/Now That's What I Call Music 15/06. Bounce.flac
deleting Various artists/Now That's What I Call Music 15/05. Gangsta Nation.flac
deleting Various artists/Now That's What I Call Music 15/04. Holidae In.flac
deleting Various artists/Now That's What I Call Music 15/03. Stand Up.flac
deleting Various artists/Now That's What I Call Music 15/02. Toxic.flac
deleting Various artists/Now That's What I Call Music 15/01. It's My Life.flac
deleting Various artists/Now That's What I Call Music 15/
deleting Various artists/Moochin' Abouts Stateside Hitlist 1960/09. Wonderful World.mp3
deleting Various artists/Moochin' Abouts Stateside Hitlist 1960/
deleting Various artists/Midnight Clear/MidightClear.jpg
deleting Various artists/Midnight Clear/10. Conventy Carol.flac
deleting Various artists/Midnight Clear/09. Lo, How A Rose E'er Blooming.flac
deleting Various artists/Midnight Clear/08. O Little Town of Bethlehem.flac
deleting Various artists/Midnight Clear/07. All My Heart This Night Rejoices.flac
deleting Various artists/Midnight Clear/06. On Christmas Night.flac
deleting Various artists/Midnight Clear/05. _Rise Up Shepherd and Follow.flac
deleting Various artists/Midnight Clear/04. The Holly And the Ivy.flac
deleting Various artists/Midnight Clear/03. Pat-A-Pan.flac
deleting Various artists/Midnight Clear/02. We Three Kings.flac
deleting Various artists/Midnight Clear/01. It Came Upon a Midnight Cleaer.flac
deleting Various artists/Midnight Clear/
deleting Various artists/Frozen/05. Let It Go.mp3
deleting Various artists/Frozen/03. For the First Time in Forever.mp3
deleting Various artists/Frozen/02. Do You Want to Build a Snowman_.mp3
deleting Various artists/Frozen/
deleting Various artists/Crunk And Disorderly/01 What You Gonna Do.mp3
deleting Various artists/Crunk And Disorderly/
deleting Various artists/City on a Hill/32 This Road.mp3
deleting Various artists/City on a Hill/
deleting Various artists/Celtic Wonder/12. The Lakes of Pontchartrain.flac
deleting Various artists/Celtic Wonder/11. Dúlamán.flac
deleting Various artists/Celtic Wonder/10. Siúil a Rún.flac
deleting Various artists/Celtic Wonder/09. The Briar and the Rose.flac
deleting Various artists/Celtic Wonder/08. The Benedy Glen.flac
deleting Various artists/Celtic Wonder/07. The Jug of Punch.flac
deleting Various artists/Celtic Wonder/06. The Bantry Girl's Lament.flac
deleting Various artists/Celtic Wonder/05. The Water Is Wide.flac
deleting Various artists/Celtic Wonder/04. The Flower of Magherally.flac
deleting Various artists/Celtic Wonder/03. Green Grow the Rushes Oh.flac
deleting Various artists/Celtic Wonder/02. The Maid That Sold Her Barley.flac
deleting Various artists/Celtic Wonder/01. An Páistín Fionn.flac
deleting Various artists/Celtic Wonder/
deleting Various artists/A Time to Remember, Vol. 1/17. Tie Me Kangaroo Down Sport.mp3
deleting Various artists/A Time to Remember, Vol. 1/

Number of files: 6,213 (reg: 4,935, dir: 1,278)
Number of created files: 821 (reg: 804, dir: 17)
Number of deleted files: 108 (reg: 86, dir: 22)
Number of regular files transferred: 1,479
Total file size: 41.25G bytes
Total transferred file size: 1.71G bytes
Literal data: 1.71G bytes
Matched data: 0 bytes
File list size: 196.59K
File list generation time: 0.044 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 1.71G
Total bytes received: 28.24K

sent 1.71G bytes  received 28.24K bytes  11.14M bytes/sec
total size is 41.25G  speedup is 24.12

luckyback or rysnc is deleting everything inside 'Various artists' for reasons I do not at all understand. It should only be deleting files in the destination if they do not exist in the source, that's what I have the options set to anyway. I also noticed at bottom portion of this log that it has created a lot of files, 804 regular files and 17 directories to be exact. I haven't added a single new file in the source so why is it creating, or re-creating a lot of the same files in the destination over again. Also, there is alot more than 804 files and 17 directories so it's like it's just picking out what it wants? Any ideas?
I was just using rsyn to start with but it would take over an hour to sync even if I had only added one new file which lead me to believe that rsync was overwritting the whole SD card everytime I synced it rather than just adding/removing what had changed. Thanks for any and all replies.

[rproffitt]

Sorry if I was unclear. My question was " how is the card mounted?"

Check my supplied link about another mount parameter.

[lewashby]

My system didn't recognise iocharset or the shortname options. I have now ran this command on my system ->
$ rsync -vrc --delete --ignore-existing --exclude ".*/" /home/me/media/music/ /media/me/MEDIA-SD/ && echo "DONE"

Now the tail end of this command looks like this.

irectory (2)
rsync: mkstemp "/media/me/MEDIA-SD/Various Artists/O Brother, Where Art Thou_/.10. Didn't Leave Nobody But The Baby (O Brother, Where Art Thou_ Soundtrack).mp3.3olHWE" failed: No such file or directory (2)
default_perms_for_dir: sys_acl_get_file(Various Artists/One Hit Wonders, ACL_TYPE_DEFAULT): No such file or directory, falling back on umask
rsync: mkstemp "/media/me/MEDIA-SD/Various Artists/One Hit Wonders/.04. Rockin' Little Angel.mp3.u5o7ya" failed: No such file or directory (2)
default_perms_for_dir: sys_acl_get_file(Various Artists/Platinum Christmas, ACL_TYPE_DEFAULT): No such file or directory, falling back on umask
rsync: mkstemp "/media/me/MEDIA-SD/Various Artists/Platinum Christmas/.07 Little Drummer Boy.mp3.BIvwgG" failed: No such file or directory (2)
default_perms_for_dir: sys_acl_get_file(Various Artists/Pop a Paris - Rock N' Roll and Mini Skirts Vol.1, ACL_TYPE_DEFAULT): No such file or directory, falling back on umask
rsync: mkstemp "/media/me/MEDIA-SD/Various Artists/Pop a Paris - Rock N' Roll and Mini Skirts Vol.1/.03. Ces bottes sont faites pour marcher.mp3.Z0ms7b" failed: No such file or directory (2)
default_perms_for_dir: sys_acl_get_file(Various Artists/Rain Man _ Original Motion Picture Sound, ACL_TYPE_DEFAULT): No such file or directory, falling back on umask
rsync: mkstemp "/media/me/MEDIA-SD/Various Artists/Rain Man _ Original Motion Picture Sound/.01 Iko Iko.mp3.sG3n3H" failed: No such file or directory (2)
default_perms_for_dir: sys_acl_get_file(Various Artists/Rain Man _ Original Motion Picture Soundtrack, ACL_TYPE_DEFAULT): No such file or directory, falling back on umask
rsync: mkstemp "/media/me/MEDIA-SD/Various Artists/Rain Man _ Original Motion Picture Soundtrack/.01. Iko Iko.mp3.R5Gc7d" failed: No such file or directory (2)
default_perms_for_dir: sys_acl_get_file(Various Artists/Sentimentally Scottish, ACL_TYPE_DEFAULT): No such file or directory, falling back on umask
rsync: mkstemp "/media/me/MEDIA-SD/Various Artists/Sentimentally Scottish/.Scottish Soldier.mp3.yVB2jK" failed: No such file or directory (2)
default_perms_for_dir: sys_acl_get_file(Various Artists/The Walking Dead_ AMC Original Soundtrack, Vol. 2, ACL_TYPE_DEFAULT): No such file or directory, falling back on umask
rsync: mkstemp "/media/me/MEDIA-SD/Various Artists/The Walking Dead_ AMC Original Soundtrack, Vol. 2/.04. Be Not So Fearful.mp3.rJTnDg" failed: No such file or directory (2)
default_perms_for_dir: sys_acl_get_file(Various Artists/White Christmas, ACL_TYPE_DEFAULT): No such file or directory, falling back on umask
rsync: mkstemp "/media/me/MEDIA-SD/Various Artists/White Christmas/.01. White Christmas.flac.4XHO4M" failed: No such file or directory (2)
rsync: mkstemp "/media/me/MEDIA-SD/Various Artists/White Christmas/.02. Jingle Bells.flac.g00rSj" failed: No such file or directory (2)
rsync: mkstemp "/media/me/MEDIA-SD/Various Artists/White Christmas/.03. Sleigh Ride.flac.UjQdZQ" failed: No such file or directory (2)
rsync: mkstemp "/media/me/MEDIA-SD/Various Artists/White Christmas/.04. Toyland.flac.QQ4Xoo" failed: No such file or directory (2)
rsync: mkstemp "/media/me/MEDIA-SD/Various Artists/White Christmas/.05. A Holly Jolly Christmas.flac.5Ykr8V" failed: No such file or directory (2)
rsync: mkstemp "/media/me/MEDIA-SD/Various Artists/White Christmas/.06. Frosty the Snowman.flac.VaC1au" failed: No such file or directory (2)
rsync: mkstemp "/media/me/MEDIA-SD/Various Artists/White Christmas/.07. Winter Wonderland.flac.oluHt2" failed: No such file or directory (2)
rsync: mkstemp "/media/me/MEDIA-SD/Various Artists/White Christmas/.08. It's Beginning to Look a Lot Like Christmas.flac.hdIT1A" failed: No such file or directory (2)
rsync: mkstemp "/media/me/MEDIA-SD/Various Artists/White Christmas/.09. O Little Town of Bethlehem.flac.FyVOP9" failed: No such file or directory (2)
rsync: mkstemp "/media/me/MEDIA-SD/Various Artists/White Christmas/.10. Deck the Halls.flac.BAAnTI" failed: No such file or directory (2)

[rproffitt]

Another idea. Since the file system on cards is well, what it is, you find you have to format that file system once in a while. I see this on my car camera since again, not a great file system.

Also, as to your last log dump, what happens when you manually copy one of those failed files over?

[lewashby]

I reformatted the sd card and ry-synced everything again, same problem. I also manually copied the files from my library to the sd card and it worked fine, go figure. Re-synced again with luckybackup and delete all the files on the destination again :(.

P.S. Why is the post marked 'Featured'?

[Gribouillis]

Did you try using a case sensitive filesystem such as ntfs ?

[rproffitt]

Since you use luckybackup and it has been left with known bugs at http://luckybackup.sourceforge.net/bugs.html I think it is time to find another sync app. However it is open source and you are free to try to fix this.

[Gribouillis]

@rproffitt Luckybackup invokes rsync, and it can display the rsync command line that it uses. It means that one can use luckybackup as a GUI tool to figure out the relevant command line options for rsync. It means that the question is not luckybackup vs some other tool, but rather rsync vs some other tool.

[rproffitt]

@G. Yes, and the file system + rsync is not playing ball. Time to try another sync app since a manual copy works. While rsync is open source, it's the OP's choice to dive into the code to find out why. Here, I continue to run into such things and over the years learned to change to other tools when a tool doesn't work for me and I don't feel like opening the source.

[lewashby]

I'm not near good enough to open the source of software and fix, change, or even understand anything. I can write very simple bash, python, php, c & c++ programs. php is the only thing I can realistically develop to any usability, and even then it's pretty minimal.

No ntfs, OS is ext4 and micro sd is fat32.

I fixed it but I still don't understand what was happening. Something really weird was going on with the music file that seems to have been causing this in some way. I had the song 'Tie Me Kangaroo down sport by Rolf Harris, album The Hits of Pop & Doo Wop, Vol. 5' downloaded to my music library, I got the song from Amazon's digital music store. Of my two directories that luckybackup/rsync kept getting mixed up were 'Various Artists & Various artists' the kangaroo song was the only song that was physically in the lowercase directory but showed in uppercase as two identical files on my desktop music player (Banshee). When I would right click on each of the two copies and click 'open in folder' one would be in the uppercase directory (Various Artists) and the other would be in the lowercase directory (Various artists), when when I would right click on the one in the lowercase directory and delete the file it would delete both copies? I have no idea! I re-download the song from amazon and this time I edited the meta data and changed the Artists field from 'Various artists' to 'Various Artists' before adding it to my library. Synced fine, no more problems. The two music files seem to be linked but in different directories with different meta data, I don't get it, never seen anything like it. And why that's been getting luckybackup/rsync confused when it comes to syncing the two Various aArtists directories I'll never know. What a cluster. Thanks everyone.

[rproffitt]

That's your best clue -> "two identical files on my desktop".

Microsoft's implementation of long file names is well, a hack that is one for the history books. You found what it was.

[lewashby]

@rproffitt what is the microsoft relevance here?

[rproffitt]

@Lewashby. I'm glad you asked. FAT32 is a Microsoft design.
In case you don't believe me.

https://en.wikipedia.org/wiki/File_Allocation_Table writes:

In order to overcome the volume size limit of FAT16, while at the same time allowing DOS real mode code to handle the format, Microsoft designed a new version of the file system, FAT32,

MSFT had many opportunities to break with the past but kept trying to keep older apps and filesystems alive and you see issues like yours when we use that file system.