Throttling TAR disk IO

[PacketMan]

When I run tar(1) to read files on one disk, and write the .tar to another disk the IO is pretty much 100%. If I wanted to throttle that, say for example 50%, how would I do it? I am using tar(1) inside a script. Any particular URLs someone can refer me to?

Thanks again everyone.

 

[jb_fvwm2]

I'd switch to rsync: --bwlimit=700 [ slow background ] ... --bwlimit=7000 [ almost full speed ]

 

[PacketMan]

Yeah I was looking at it, scrolling down through its various options, but I not considering using it yet. Looks like TAR(1) doesn't have any options to control IO usage.

 

[T-Daemon]

There is this article, using sysutils/pv to slow down tar:

Slowing down (limiting) tar, mysqldump or other processes to save IO bandwidth | Closer to Code

Sometimes we want to perform some sort of tasks that consume whole available IO bandwidth. This may lead to some unexpected behaviours from our OS. OS might even kill the given process due to resource lack. Lets take an example. We want to tar.gz a huge directory with a lot of files in it. Our

mensfeld.pl

 

[ralphbsz]

I don't think you can. Individual programs can limit how much IO they do; jb_fvwm2 gave an example for rsync. There is no OS-wide facility for limiting/throttling the IO of a particular process, nor is there an IO priority system and scheduler. For CPU usage, there are such things as priorities, the nice command, the whole mechanism behind it, not for IO.

But let me ask another question: Why do you care? What's wrong with tar using 100% of the disk? Matter-of-fact, you have to remember that spinning disks are one of the few things in the universe that get more efficient the more overloaded they are. From an overall throughput standpoint, you really want your tar to run as fast as possible, even if that's an inconvenience for other processes that use the same disk.

 

[T-Daemon]

using sysutils/pv to slow down tar:

Here is another one: misc/cstream , cstream(1).

Backup: bandwidth throttling with cstream (#4524) · Issues · ISPConfig / ISPConfig 3 · GitLab

I noticed last night that my disk IO usage was 100% and server was hardly working. I saw the ispconfig backup is to blame. To fix...

git.ispconfig.org

 

[PacketMan]

But let me ask another question: Why do you care? What's wrong with tar using 100% of the disk?

Well, perhaps I have other files on one of those disks and using those files become sluggish. Yes I would normally run TAR after hours, but sometimes I don't or can't.

 

[eldaemon]

Sure, you can throttle tar. Put something in a pipe with it.

tar cf - . | slowitdown > foo.tar

slowitdown could be xz or an actual pipe throttler. You can use pv to show the speed and verify if something is indeed slowing it down. But note that you would want to put pv before the compressor and not after.

 

[eldaemon]

There is this article, using sysutils/pv to slow down tar:

Slowing down (limiting) tar, mysqldump or other processes to save IO bandwidth | Closer to Code

Sometimes we want to perform some sort of tasks that consume whole available IO bandwidth. This may lead to some unexpected behaviours from our OS. OS might even kill the given process due to resource lack. Lets take an example. We want to tar.gz a huge directory with a lot of files in it. Our

mensfeld.pl

Ah, nice! I had no idea pv supported that. Definitely the way to go for a pipe-based solution.

 

[PacketMan]

Thanks everyone.