Function fallocate() & splice() from FreeBSD.

nagual2

New Member


Messages: 18

Some time ago I began to develop a test file systems for Linux and FreeBSD
https://github.com/nagual2/Test

And to assess the effects of file fragmentation on performance file system I needed to implement add and remove blocks in the middle of the file - "hole punching" I can't find an analogue functions fallocate() and splice() under FreeBSD. And while I'm running linux emulator these functions are not available when compiling:
Code:
# if HAVE_LINUX_FALLOCATE
return fallocate (fd, mode, offset, len);
# else
return EIO_ENOSYS ();
# endif
Function posix_fallocate() not analgichna fallocate().
How do I implement fallocate() and splice() on FreeBSD? Or how to use those that are in the emulator linux?
 

expl

Aspiring Daemon

Reaction score: 126
Messages: 664

You can always implement these syscalls in userworld, but then it would be unfair to compare them to Linux since obviously due to design they will run much slower. In conclusion you should not write I/O tests that will target multiple OSes using Linux only not portable syscalls...
 
OP
nagual2

nagual2

New Member


Messages: 18

I am developing a test with Perl using library IO::AIO is implemented as an interface to libeio (http://software.schmorp.de/pkg/libeio.html). But it turned out that FreeBSD does not work. Do I understand correctly that there are no analogues FreeBSD fallocate() and splice()? As for the file system ZFS and UFS UFS2 admit they add - remove the block of any length with any offset in the file, or there are restrictions on reversal of file systems?
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 9,332
Messages: 33,894

nagual2 said:
Do I understand correctly that there are no analogues FreeBSD fallocate() and splice()?
Why would we need them?

This is a nonportable, Linux-specific system call. For the portable,
POSIX.1-specified method of ensuring that space is allocated for a file, see
posix_fallocate().
http://www.kernel.org/doc/man-pages/online/pages/man2/fallocate.2.html

Same for splice():
CONFORMING TO
This system call is Linux-specific.
http://www.kernel.org/doc/man-pages/online/pages/man2/splice.2.html
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 9,332
Messages: 33,894

I understood that, I'm just wondering why FreeBSD would need them.
 
OP
nagual2

nagual2

New Member


Messages: 18

And how can you add a block in the middle of the file length of 20 gigabytes? Without fallocate()? Without the "hole punching"? In the middle of the file, not the end.
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 9,332
Messages: 33,894

Like you would solve it without those functions, copy the data until the point you need to insert, insert the data and copy the rest of the original file. Delete old data.
 

expl

Aspiring Daemon

Reaction score: 126
Messages: 664

Well FreeBSD has ftruncate() syscall, thats is very similar and differently from fallocate is portable.
 
OP
nagual2

nagual2

New Member


Messages: 18

expl said:
Well FreeBSD has ftruncate() syscall, thats is very similar and differently from fallocate is portable.
function ftruncate() truncates the end of the file, it does not set the length of the deleted fragment.
Сan not be done splice() from ftruncate().
 

expl

Aspiring Daemon

Reaction score: 126
Messages: 664

nagual2 said:
It's impossible. File sizes over 10 gigabytes.
It is possible, just need to do it in reverse order. Allocate space at the end of the file equal to the size of your 'hole', then move data from before the allocation bit by bit (use buffered and not raw read/write functions, they are optimized for this kind of thing) towards the end until you reach the position where you need to insert the data, write the change and quit.

edit:
I am pretty sure thats how fallocate works, it's just done at kernel level so it has advantage of knowing exact block layout so it can optimized much better.
 
OP
nagual2

nagual2

New Member


Messages: 18

This is possible but it is very slow and it's creators extra unnecessary load on the drive.
 

expl

Aspiring Daemon

Reaction score: 126
Messages: 664

nagual2 said:
This is possible but it is very slow and it's creators extra unnecessary load on the drive.
Can you better explain why would you test this sort of thing? No well designed software will ever need/use such behavior so why test for it?
 
OP
nagual2

nagual2

New Member


Messages: 18

To quickly create test conditions performance file system under the maximum fragmentation are two options - one file large file and many small. So if one more - provide read and write operations on the test disk from 0% filled with files up to 100% and calculate the average read and write speeds. As soon as the read and write speeds stop falling, we believe the maximum fragmentation and the resulting rate for the FS. Actually more complicated: If we write then write in the middle of inserting a file with an arbitrary offset from the start and arbitrary length unit. If you remove the block then do not remove it completely and cut out of the middle with an arbitrary displacement and arbitrary length and then add it to the end. What are the functions that will allow it to implement?

Any ideas?
 

expl

Aspiring Daemon

Reaction score: 126
Messages: 664

nagual2 said:
To quickly create test conditions performance file system under the maximum fragmentation are two options - one file large file and many small. So if one more - provide read and write operations on the test disk from 0% filled with files up to 100% and calculate the average read and write speeds. As soon as the read and write speeds stop falling, we believe the maximum fragmentation and the resulting rate for the FS. Actually more complicated: If we write then write in the middle of inserting a file with an arbitrary offset from the start and arbitrary length unit. If you remove the block then do not remove it completely and cut out of the middle with an arbitrary displacement and arbitrary length and then add it to the end. What are the functions that will allow it to implement?

Any ideas?
You can't achieve accurate maximum fragmentation from userworld, actual fragmentation will vary a lot depending on filesystem and physical media. But if you want to simulate fragmentation anyways you have to use many files to make it as accurate and portable as possible.
 
OP
nagual2

nagual2

New Member


Messages: 18

I understand that the length of the recording unit should be a multiple of the block size file system.
 
Top