why ncdu and du -sch show different size in FreeBSD and Linux

mfaridi

mfaridi

I have one folder in Ubuntu and move it to FreeBSD Box with scp and run ncdu
I see this in Linux Box
Code: 
  271.1MiB  /portabl
and I see this in FreeBSD box
Code: 
238.2MiB [##########] /portable
So I run du -sch in Linux Box and I see this
Code: 
272M	portable/
272M	total
and I run du -sch in FreeBSD and I see this
Code: 
238M	portable
238M	total
why I see different size of one folder in FreeBSD and Linux?
 
Any hidden files or directories on the Linux box that might have been missed by the scp?

What filesystem on the Linux box? What blocksize is it using?
 
ckester said:
Any hidden files or directories on the Linux box that might have been missed by the scp?

What filesystem on the Linux box? What blocksize is it using?
Click to expand...

In Linux I use Ext4 and when I run this command df -T , I see this
Code: 
              ext4     7583436   1447540   5750668  21% /
none      devtmpfs      248088       204    247884   1% /dev
none         tmpfs      254064         0    254064   0% /dev/shm
none         tmpfs      254064        32    254032   1% /var/run
none         tmpfs      254064         0    254064   0% /var/lock
none       debugfs     7583436   1447540   5750668  21% /var/lib/ureadahead/debugfs
/dev/sda1     ext2      233191     19949    200801  10% /boot
m
and when I run this command mount I see this
Code: 
/dev/mapper/ubuntu-root on / type ext4 (rw,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
none on /sys type sysfs (rw,noexec,nosuid,nodev)
fusectl on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
none on /dev type devtmpfs (rw,mode=0755)
none on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
none on /dev/shm type tmpfs (rw,nosuid,nodev)
none on /var/run type tmpfs (rw,nosuid,mode=0755)
none on /var/lock type tmpfs (rw,noexec,nosuid,nodev)
none on /var/lib/ureadahead/debugfs type debugfs (rw,relatime)
/dev/sda1 on /boot type ext2 (rw)
 
Pushrod said:
do this on both:

[CMD=""]find . -type f | perl -nle '$s+=-s; END {print $s}'[/CMD]
Click to expand...

when I run your command in linux I see this
Code: 
229365985
and when I run it in FreeBSD I see this
Code: 
214845083
they are different
 
If you list the files on each machine with # ls -al is there any difference in file sizes reported in the results?

You should also try # ls -as to see if there is any difference in the number of filesystem blocks used for each file.

I'm not a filesystem expert (*), and don't know if this is even possible on Unix systems,
but I'm wondering if perhaps what you're seeing is:
(a) the difference between a somewhat fragmented disk and an unfragmented one; or
(b) a fundamental difference in the efficiency with which the two filesystems use diskspace.

I.e, the difference between

[xxx ][xxx ][xxx ][xxx ]
and
[xxxx][xxxx][xxxx][ ]

If you count x's they're the same.
If you count [] blocks used, they're different.

(* It sounds like you really need to get an answer, from someone who knows what they're talking about. Sorry, that's not me. You might try asking the freebsd-fs@freebsd.org mailing list instead.)
 
It sounds like hidden files to me, too.
% ls -a | wc -l
is probably different on the two systems.
 
wblock said:
It sounds like hidden files to me, too.
% ls -a | wc -l
is probably different on the two systems.
Click to expand...

I run that command in FreeBSD and I see this
Code: 
narges ~/jadi/portable [433] ls -a | wc -l                                                                                                 10
and I run thay command In Linux and I see this
Code: 
mostafa@ubuntu:~/portable$ ls -a | wc -l
10
FreeBSD and Linux show me one thing
 
ckester said:
If you list the files on each machine with # ls -al is there any difference in file sizes reported in the results?

You should also try # ls -as to see if there is any difference in the number of filesystem blocks used for each file.

I'm not a filesystem expert (*), and don't know if this is even possible on Unix systems,
but I'm wondering if perhaps what you're seeing is:
(a) the difference between a somewhat fragmented disk and an unfragmented one; or
(b) a fundamental difference in the efficiency with which the two filesystems use diskspace.

I.e, the difference between

[xxx ][xxx ][xxx ][xxx ]
and
[xxxx][xxxx][xxxx][ ]

If you count x's they're the same.
If you count [] blocks used, they're different.

(* It sounds like you really need to get an answer, from someone who knows what they're talking about. Sorry, that's not me. You might try asking the freebsd-fs@freebsd.org mailing list instead.)
Click to expand...

when i run ls -la in Linux I see this
Code: 
total 48
drwxr-xr-x 6 mostafa mostafa  4096 2010-12-30 16:26 .
drwxr-xr-x 7 mostafa mostafa  4096 2010-12-30 16:25 ..
drwxr-xr-x 4 mostafa mostafa  4096 2011-01-02 18:41 db
drwxr-xr-x 3 mostafa mostafa  4096 2010-12-30 16:25 etc
drwxr-xr-x 2 mostafa mostafa 12288 2011-01-02 16:57 pkg
-rwxr-xr-x 1 mostafa mostafa   826 2010-12-30 16:25 powerpill
-rwxr-xr-x 1 mostafa mostafa   129 2010-12-30 16:25 pp
-rwxr-xr-x 1 mostafa mostafa  1461 2010-12-30 16:26 README
-rwxr-xr-x 1 mostafa mostafa   254 2010-12-30 16:25 sync
drwxr-xr-x 4 mostafa mostafa  4096 2010-12-30 16:25 Xyne
and when I run ls -as in Linux I see this
Code: 
total 48
 4 .   4 ..   4 db   4 etc  12 pkg   4 powerpill   4 pp   4 README   4 sync   4 Xyne

=======================================================================
when I run ls -la in FreeBSD , I see this
Code: 
total 28
drwxr-xr-x  6 narges  narges   512 Dec 30 12:39 ./
drwxr-xr-x  4 narges  narges  5120 Jan  1 12:21 ../
-rwxr-xr-x  1 narges  narges  1461 Dec 31 15:20 README
drwxr-xr-x  4 narges  narges   512 Dec 30 12:39 Xyne/
drwxr-xr-x  4 narges  narges   512 Dec 30 12:39 db/
drwxr-xr-x  3 narges  narges   512 Dec 30 12:39 etc/
drwxr-xr-x  2 narges  narges  4608 Dec 30 12:46 pkg/
-rwxr-xr-x  1 narges  narges   826 Dec 30 11:36 powerpill
-rwxr-xr-x  1 narges  narges   129 Dec 31 15:20 pp
-rwxr-xr-x  1 narges  narges   254 Dec 31 15:20 sync
and when I run ls -as in FreeBSD , I see this
Code: 
total 28
 2 ./          6 ../         2 README      2 Xyne/       2 db/         2 etc/        6 pkg/        2 powerpill   2 pp          2 sync
========================================================================
they are different output in FreeBSD and Linux'
 
For the time being, disregard the results for the directories in the lists. Look at the results for the files. The file sizes are the same, while the block counts are different. What does that tell you?

Different filesystems, apparently different blocksizes, and probably different levels of fragmentation too.

What motivates your question in this thread? Are you worried that you didn't get a complete copy when you did the scp?

From what I see here, I think you got a complete copy. But du probably isn't the best way to verify that. A total checksum of the all the files copied would be better.
 
If you must know, do a diff.
[CMD=""]diff -rq srcdir destdir[/CMD]

The output from the Perl command should have been the same. I think you did NOT get a complete copy. Use rsync or something to sort this out.
 
Assuming that Pushrod's perl command sums up the file sizes, then I agree, you didn't get a complete copy.

If you haven't already solved the problem, here's one way to track down the missing or incomplete files:

Run this command on each system, in the top directory you used for the scp:

$ find . -type f -ls | sort -k11 | awk '{print $7 "\t" $11}' > /tmp/list

Then copy the resulting list from the Linux system to the FreeBSD system and compare them:

$ diff /tmp/list.linux /tmp/list.freebsd

The two lists contain the filesize and pathname for each file found.

If all of the files were copied and are the same size on each system, the two list files will be the same and diff should exit without printing any output.

If a file was incompletely transferred, you'll see it listed twice in the diff output, with different filesize numbers.

If it wasn't transferred at all, you'll see it only once in the diff output, with a < as the first character on the line.

If some of the files on your Linux machine are symbolic links, they might have been scp'ed to your FreeBSD box as regular files with size zero. Each of them will show up only once in the diff output, because the find command I gave you only lists regular files, not symbolic links. In this case, the first character on the diff output line will be a >, indicating that the file was found on the FreeBSD box.

(BTW, that's an eleven, not two L's, in the sort and awk commands above.)
 
The command find . -type f | perl -nle '$s+=-s; END {print $s}' adds the size of each file to "$s". Just before Perl exits, it prints the value of $s.

"-s" in Perl is analogous to "-s" in /bin/test.
 
ckester said:
Assuming that Pushrod's perl command sums up the file sizes, then I agree, you didn't get a complete copy.

If you haven't already solved the problem, here's one way to track down the missing or incomplete files:

Run this command on each system, in the top directory you used for the scp:

$ find . -type f -ls | sort -k11 | awk '{print $7 "\t" $11}' > /tmp/list

Then copy the resulting list from the Linux system to the FreeBSD system and compare them:

$ diff /tmp/list.linux /tmp/list.freebsd

The two lists contain the filesize and pathname for each file found.

If all of the files were copied and are the same size on each system, the two list files will be the same and diff should exit without printing any output.

If a file was incompletely transferred, you'll see it listed twice in the diff output, with different filesize numbers.

If it wasn't transferred at all, you'll see it only once in the diff output, with a < as the first character on the line.

If some of the files on your Linux machine are symbolic links, they might have been scp'ed to your FreeBSD box as regular files with size zero. Each of them will show up only once in the diff output, because the find command I gave you only lists regular files, not symbolic links. In this case, the first character on the diff output line will be a >, indicating that the file was found on the FreeBSD box.

(BTW, that's an eleven, not two L's, in the sort and awk commands above.)
Click to expand...

wow great
I run that perl command in both of system Linux and FreeBSD and move output to one directory and run this command
Code: 
 diff list.linux freebsd.list
and I see this. I attach txt file, it is big file, how much they are different, I want use rsysnc for this problem
 

Attachments

You must log in or register to reply here.
Back
Top