Shell Easiest way to move files up in directory hieararchy?

ondra_knezour

Aspiring Daemon

Reaction score: 225
Messages: 797

I have thousands files in hierarchy like /somewhere/year/month/day/hour/files. Also files are named accordingly, nfcapd.yearmonthdayhourminute. In both cases all one number data (month, day, hour) are padded with zero to two places, for example 2020/01/01/01/nfcapd.2020010101xx (for minutes). And now I need to move all of them one step in filesystem hierarchy up, effectively eliminating the hour directory in the structure. How would you do it, preferably using shell without installing any additional software?
 

Zirias

Daemon

Reaction score: 1,346
Messages: 2,370

Should be what the second form of mv(1) does:
Code:
SYNOPSIS
     mv [-f | -i | -n] [-hv] source target
     mv [-f | -i | -n] [-v] source ... directory

So, just use mv * ..
 
OP
O

ondra_knezour

Aspiring Daemon

Reaction score: 225
Messages: 797

Yes, this would be easy, until you start counting :)

several sources x 2 years x 12 month x 30 days x 24 hours -> looks like 35000 directories in my case, no way to cd & mv and stay sane
 

Zirias

Daemon

Reaction score: 1,346
Messages: 2,370

So, you want to do this for many of these "hour" directories at once? Might be a job for find(1), something similar to find . -execdir mv \{\} .. \+ from the parent of all these hour dirs… (untested, you might want to use echo instead of mv first to see whether the command will do the right thing)
 

sko

Aspiring Daemon

Reaction score: 348
Messages: 623

if all files that are to be moved are named 'nfcapb.*' something like find . -name 'nfcapd.*' -execdir mv {} ../ \; in the base directory of your hierarchy should work.

edit:
explanation: 'execdir' calls the command inside the directory of the match and {} is replaced with the match.
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 11,625
Messages: 37,973

You probably want to output a list of files and pipe it through xargs(1) in batches (trying to do everything in one go is likely to cause a "too many parameters" or "line exceeded" error). Using find(1) in combination with single action -exec mv is going to be a terribly slow process.
 

Zirias

Daemon

Reaction score: 1,346
Messages: 2,370

You probably want to output a list of files and pipe it through xargs(1) in batches. Using find(1) in combination with single -exec mv is going to be a terribly slow process.
Not if you use the version with +, this will add as many files as possible to the commandline (just like xargs).
 

VladiBG

Daemon

Reaction score: 510
Messages: 1,112

sko i like your method to solve this. I was thinking of shell script with loops but your way is much simple.
Bash:
#!/bin/sh
for y in $(seq 2000 1 2022); do
    for m in $(seq -f "%02g" 01 1 12); do
        for d in $(seq -f "%02g" 01 1 31); do
            for h in $(seq -f "%02g" 01 1 24); do
                for file in /home/user/test/$y/$m/$d/$h/*; do
                    echo "mv $file .."
                done
            done
        done
    done
done

Also you will need another step to remove the empty directories after that....
 

sko

Aspiring Daemon

Reaction score: 348
Messages: 623

You probably want to output a list of files and pipe it through xargs(1) in batches (trying to do everything in one go is likely to cause a "too many parameters" or "line exceeded" error). Using find(1) in combination with single action -exec mv is going to be a terribly slow process.
Yes, find(1) is quite inefficient in many ways - that's why you usually avoid it like the plague for scripting. But for one-off operations it usually doesn't matter if it takes 30 seconds or 30 minutes.
Oftentimes you spend way more time developing (and debugging) the 'most efficient way'™ than just going with the simpler, slower solution you had at the beginning.
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 11,625
Messages: 37,973

But for one-off operations it usually doesn't matter if it takes 30 seconds or 30 minutes.
Oh for sure. I much rather have things go slow and steady than fast to hell any time ;)
 

Zirias

Daemon

Reaction score: 1,346
Messages: 2,370

Yes, find(1) is quite inefficient in many ways - that's why you usually avoid it like the plague for scripting.
What's inefficient is using -exec ... {} ... ;, passing one found file at a time to a command to execute. -exec ... {} ... + solves this issue, replacing {} with as many names as possible.

Anything else that's inefficient with find(1)? :-/
 
OP
O

ondra_knezour

Aspiring Daemon

Reaction score: 225
Messages: 797

Thank you all, -execdir was what I was missing to put it all together on my own :)

It looks like {}+ has to be at the end of the command, so it is not usable with mv, but even with semicolon and manual hoping between directories I am almost done in less then half hour.
 
Top