Other Formatting a Hard Disk Drive

[Terry Mester]

Disconnecting and reading/writing a disk in a case is not difficult; all it takes is bringing a small laptop, android phone or Raspberry Pi with a USB to SATA adapter and a cable.

If you are going to participate in a Forum Thread, then it is your duty to read all the preceding Posts, and to not distort the statements of others. I already commented on this in Post #13.

What does "authentic" mean? If you mean that it contains the original Dennis and Ken source code: As far as I know, that is today only available in the form of AIX and HP-UX, ...

When I refer to 'authentic' UNIX I don't mean the original programme code which is not relevant, but rather I'm referring to original 'features', including security features, of UNIX. FreeBSD has deviated from some of UNIX's features. Some of FreeBSD's changes are beneficial: the abolition of the Modify and Append Attributes, the addition of the Link attribute, the switching of the Wheel / Superuser Group Number from 255 to 0, and the addition of more Groups / Users beyond the original 256. However, some of BSD's changes are not beneficial, and have degraded FreeBSD's usefulness as a 'networking' Operating System: namely, the abolition of the Regular Superuser and the Group Leader User. The abolition of these two important User categories unfortunately renders FreeBSD inferior to UNIX for networking -- whether it be a School or Business setting. It would be very beneficial for FreeBSD to restore these two User categories.

You say that that division among the BSDs prevents software manufacturer from offering software for BSD. What software manufacturers? The amount of commercial software that is produced for any BSD is minuscule to begin with. ...

Ah, one good example is the US Robotics Internal Modem which I wanted to buy for use with FreeBSD, but they don't provide a Driver for BSD! They do however provide Drivers for Linux and MAC! The fact that the amount of commercial software for BSD is "minuscule" is a BIG PROBLEM if you expect Computer users to switch to FreeBSD from Windows or Linux. The problem with schism in the BSD community makes BSD look like a fringe group, and this is a reason why software is not produced for FreeBSD!

Your comment about OpenBSD is nonsense. Yes, it's community is smaller than FreeBSD. ... OpenBSD is all about auditing every line of code, reducing the amount of code, and maintaining very high quality, at the expense of features. The people who are interested in that could not work on FreeBSD, as the philosophy of that OS is different. So let them work on their favorite project; they are making a very good (but very different) product too. Before you make incendiary and nonsensical statements telling OpenBSD programmers what they should do, you really need to educate yourself on the existing goals and cultures of the projects.

You're not reading what I stated. I said that "I think OpenBSD has every right to exist, but they should not use the 'BSD' acronym which should be reserved to FreeBSD in order to avoid confusion". OpenBSD probably has less than 10,000 users which, in my opinion, makes it a waste of time to maintain, but you're welcome to disagree with my opinion. I do salute and commend them if they are zealous in quality control and reducing the amount of code. I sincerely hope that FreeBSD programmers are also concise and efficient in containing the size of their programme code. Microsoft is an example of negligent programming which has resulted in the Windows OS transforming into a Cyclops.

... But in many other situations, it would not increase security at all, because other gaps (such as physical disconnect of drive or booting from another drive) remain wide open. Simply claiming that this is "some other person's problem" (namely the BIOS manufacturer) is unrealistic. Disconnecting and reading/writing a disk in a case is not difficult; all it takes is bringing a small laptop, android phone or Raspberry Pi with a USB to SATA adapter and a cable. And what is your exact proposal for how to help the person whose single user mode password doesn't work (either because they forgot it or the passwd file has become unreadable)? Before making single used mode password mandatory, you have to come up with a reasonable solution for that, and I haven't seen any.

Again, you're not correctly reading what I'm writing. I never said that a Password for Single User mode should be "mandatory". I merely said that it should be the "default" setting at Installation which a user can then change on their own by editing the '/etc/ttys' File. I also clearly stated in my last Post that I don't even think that a Password for the 'root' Superuser should be mandatory!

That fact that someone can steal someone's Hard Drive DOES NOT mean that the Operating System should roll over, and leave itself open to hackers! Once again you're mixing up two different aspects of security: the OS and the physical security of the Computer. Physical security and the BIOS settings are the sole responsibility of the Computer owner -- NOT the Operating System. The Operating System is only responsible to prevent hacking into itself.

I did actually consider the problem of a corrupted Password File before you even mentioned it. In that circumstance the OS should refer to a backup Password File. The OS could back up the File at Bootup, and the Computer user could back it up more often as they wished. / As for someone who forgets their Password, they can temporarily replace the Password File with one containing a 'root' Password they know simply by mounting their Hard Drive onto another FreeBSD system. Absent-minded people could also set the 'toor' Password to <ENTER>.

 

[Deleted member 30996]

OpenBSD probably has less than 10,000 users which, in my opinion, makes it a waste of time to maintain, but you're welcome to disagree with my opinion.

I have 2 laptops running OpenBSD. Does that count as one user or two?

 

[Terry Mester]

I guess I'll have to re-tabulate down to 9999.

I was curious to know if you use FreeBSD on any Computer? What features of OpenBSD draw you to that over FreeBSD?

 

[Deleted member 30996]

I was curious to know if you use FreeBSD on any Computer? What features of OpenBSD draw you to that over FreeBSD?

I have 8 laptops, 6 of them Thinkpads. 2 are i386, one each OpenBSD and FreeBSD, and the rest amd64. 5 currently are running FreeBSD, 2 OpenBSD and 1 OpenIndiana. One of my FreeBSD boxen serves as my dedicated .mp3 player.

I prefer FreeBSD as a desktop OS, possibly due to my familiarity with it, and use ports exclusively for my 3rd party programs. The ports tree is updated frequently, as are security issues, and I can do everything I desire when it comes to everyday desktop activities.. I have a set number of programs I install on a regular basis that suit my needs and a desktop config to maximize my work space and style. If you've seen a screenshot of one of my FreeBSD machines you've seen them all. When FreeBSD 11.2-RELEASE comes out I'll rebuild them all at once from scratch to save time.

Having OpenBSD boxen allow me to increase my knowledge and BSD skills. It also comes in handy if there is a vulnerability that effects my FreeBSD machines and may not effect my OpenBSD boxen. The pf firewall is supposed to be more current than FreeBSD uses and it's known as the "most secure" OS. I can do anything on my OpenBSD boxen I can my FreeBSD machines, sometimes with different 3rd party programs (using pkg) and am comfortable with either. It really comes down to which machines I'd rather use or have fired up at the time as to what OS I'm using.

I use my OpenIndiana box to learn more about UNIX proper than anything. I did have an Oracle box, too, but turned it into a FreeBSD box in favor of supporting OpenIndiana.

Aside from the fact it's boring having 8 laptops running the same OS.

 

[bds]

Ah, one good example is the US Robotics Internal Modem which I wanted to buy for use with FreeBSD, but they don't provide a Driver for BSD! They do however provide Drivers for Linux and MAC! The fact that the amount of commercial software for BSD is "minuscule" is a BIG PROBLEM if you expect Computer users to switch to FreeBSD from Windows or Linux. The problem with schism in the BSD community makes BSD look like a fringe group, and this is a reason why software is not produced for FreeBSD!

The US Robotics case is indeed one good example - I'm guessing that this is a binary blob of essentially unauditable code that the user is expected to trust enough to run with kernel-mode privileges on the machine. I wonder how specific the versions of these systems must be, and how quickly the commercial driver will be rendered obsolete. If such a driver was available for FreeBSD, I wouldn't touch it anyway, but instead seek out hardware that didn't suck and could be supported directly over the anticipated lifetime of the kit.

I don't get a sense that anyone here is expecting users to flock to FreeBSD from Windows or Linux, or that maximizing some (arbitrary and imperfect) metric of popularity is a primary goal of the project. I'd also warn that the only significant effort towards operating system convergence in the last few years is the one that gave us the delectable SystemD. Actually, there is another one I can think of - Windows on a phone. Remember that?

If anybody has a claim on the "BSD" acronym it is the Regents of the University of California, Berkeley. If, as you say, OpenBSD et al shouldn't have BSD in the name then neither should FreeBSD - then BSD will revert back to the original distribution that came from Berkeley and the true purists will be happy. Whilst we're at it, the plurality of Linux distributions is also causing confusion. Lets invite Mr Torvalds to select one that can have the right to carry the Linux name. Or better still, have some kind of popularity contest once a year. I also find the number of Windows editions and licensing model somewhat Byzantine, and consider it would benefit from some simplification.

The Operating System is only responsible to prevent hacking into itself.

An operating system's only responsibility in this respect is to provide and document features that support the implementation of a security policy.
Casting hardware/firmware concerns aside as Someone Else's Problem, there's still the issue of the bootcode/loaders not also being locked down by default - I previously intimated that it makes little sense to change one link in the chain and not all the others - as has already been pointed out in this discussion, it is actually worse if it draws users to make false assumptions of the ability of a default install to resist attackers with console access at boot time.

 

[Terry Mester]

I have 8 laptops, ... The ports tree is updated frequently, as are security issues, and I can do everything I desire when it comes to everyday desktop activities.. ...
When FreeBSD 11.2-RELEASE comes out I'll rebuild them all at once from scratch to save time....
The pf firewall is supposed to be more current than FreeBSD uses and it's known as the "most secure" OS. ...

I don't know how susceptible FreeBSD is to Internet Bots which are largely focused on Microsoft Windows. (I had suffered many infections on Windows 2000.) Have you given any thought to only connecting one of your Computers to the Modem, and then have that Computer act as a Server for the other seven. This should protect the others from being infected by Bots. I now use a second Slave Hard Drive in my Desktop for storing all my personal Files from both Windows and BSD. If the OS screws up I can easily re-install it on the Master Drive without affecting my Files.

Rather than upgrading to V11.2 it might be better to wait for the new V12. Do you have any technique for saving & reinstalling your personal Files when you upgrade the OS? What I did when recently reformatting Windows XP was to save both my personal Files and what Windows calls "Application Data" for various software programmes to a Flash Drive, and then after re-installing the software, I copied my Application Data for that software back to the new Directories. This restored everything -- including Mozilla Thunderbird & Firefox -- back to where it was before suffering the Windows 'Bluescreen' horror.

...I wonder how specific the versions of these systems must be, and how quickly the commercial driver will be rendered obsolete. If such a driver was available for FreeBSD, I wouldn't touch it anyway, but instead seek out hardware that didn't suck and could be supported directly over the anticipated lifetime of the kit.

Do you not consider US Robotics to be a quality manufacturer? I am not familiar with their history, but their products appear to be old and reliable. The Drivers can be updated as Operating Systems change, but their Hardware stays the same. The Driver is only needed to manage IRQs and the Bus Clock, and with FreeBSD this code probably wouldn't change from one Version to the next. I did buy the US Robotics external Modem instead which should work with BSD via the Serial Port. I intend to ask USR for the source code for their MS & Linux Drivers in order to try to work with the BSD community on producing one for FreeBSD.

I don't get a sense that anyone here is expecting users to flock to FreeBSD from Windows or Linux, or that maximizing some (arbitrary and imperfect) metric of popularity is a primary goal of the project. ...

Everyone here should desire FreeBSD to become more widespread, but that will only happen once it has much more software. UNIX / FreeBSD is the perfect OS for school settings, but lack of software is a hindrance. Schools pay a heavy licensing fee to Microsoft for using their products.

... it is actually worse if it draws users to make false assumptions of the ability of a default install to resist attackers with console access at boot time.

New users can't make any assumptions because they won't know anything about FreeBSD for a period of time. I'm simply arguing that the installed version should not leave them vulnerable before they get accustomed to it.

 

[Deleted member 30996]

I don't know how susceptible FreeBSD is to Internet Bots which are largely focused on Microsoft Windows. (I had suffered many infections on Windows 2000.) Have you given any thought to only connecting one of your Computers to the Modem, and then have that Computer act as a Server for the other seven. .

They have to infect me first. I don't surf the net with Scripting enabled and if surfing random sites or going to certain ones spoof my UserAgent to either Windows or a Mac. That in addition to having a router that performs Stateful Packet Inspection backed up by pf on each of my machines. I don't even allow remote access for myself and disallow certain service from starting in /etc/rc.conf.

I'm a member of a site whose owner isn't too picky about who they sell ad-space to. For a while it was reported that a Google red warning page popped up when you landed with a warning about Script-Based Ads downloading malware. I don't allow such silliness from Google or Mozilla. When it was reported by forum members they had been infected by script-based ads, drive-by download or what be it, it didn't worry me in the least or stop me from going there daily.

I've had a pfSense box, and need to get a card for one of my Thinkpads so I can do something similar again, but I don't really see what you mean by "then have that Computer act as a Server for the other seven" or the need. FreeBSD is primarily used as server, you know.

This should protect the others from being infected by Bots.

Then you have a MITM and all your data is sniffed.

Rather than upgrading to V11.2 it might be better to wait for the new V12. Do you have any technique for saving & reinstalling your personal Files when you upgrade the OS?

No, I'll go with 11.2 and run that till the next major release. I haven't rebuilt any of my FreeBSD laptops I originally built 11.1 on when came out last year and they all run smooth as silk.

I save my personal files to a Flash Drive, and my /etc Directory, then populate each drive from the same USB stick for redundant backup. That way I can move all my personal files to the new build and individually edit each file I need to from the old /etc Directory.

 

[Terry Mester]

... but I don't really see what you mean by "then have that Computer act as a Server for the other seven" or the need. FreeBSD is primarily used as server, you know.

I was thinking of the Computer BIOS option to boot from a Network. If you were able to boot seven of your Computers from the one connected to the Modem, then I'm guessing the BSD OS should provide some Firewall protection to the other seven. The Computer connected to the Modem would be the one assigned the IP Address.

 

[Deleted member 30996]

The Computer connected to the Modem would be the one assigned the IP Address.

My router is what gets assigned the IP#. It carries out NAT to set the LAN subnet and assigns the individual addresses in that range for each computer connected. It also serves as a firewall doing SPI.

The router also keeps track of the MAC and machine name of each device that connects to it, and keeps a permanent record of it after it has been disconnected from the LAN. When that device reconnects it gets the same IP previously assigned to it by the router.

Each machine has its own pf firewall, I use the same basic ruleset on FreeBSD and OpenBSD with only a slight change in egress syntax. If everything goes as planned, nothing should get past the router. pf is there in case it does. I'm not too worried about it.

When I first got cable they only provided the pass-thru modem and I ran my FreeBSD laptops connected directly to the internet for months without a second thought. I only got a router so I could run more than one machine online at once.

 

[Terry Mester]

... Whilst we're at it, the plurality of Linux distributions is also causing confusion. Lets invite Mr Torvalds to select one that can have the right to carry the Linux name. ...

I consider Linux to be a hopeless case of ever-increasing chaos.

 

[Terry Mester]

I got so sidetracked by the other discussions that I didn't finish the original issue relating to formatting my Slave Hard Drive. Before I execute the newfs Command I wanted to verify that I have the syntax (below) correct. Is 1% Free Space adequate, or could I use 0%?

newfs -J -U -b 4096 -f 512 -j -m 1 /dev/ada1s2a

 

[ralphbsz]

You can set -m to zero. Consider the consequences if this is the root disk, or any disk holding directories that are necessary for normal system operation (such as /etc, /var, /usr, and so on): A normal unprivileged user can then use the last byte of free space, at which point root can't write either, at which point the system is likely to crash (or at least misbehave very badly). That makes a great opening for DoS attacks (intentional and unintentional).

Furthermore consider the performance. File systems get very inefficient when they are very full. Allowing the file system to get more than 99% full will make it run very slow. In the interest of user sanity, I would leave the -m with at 5% or more, and then watch that root doesn't use up the remaining space.

In summary: I would leave it at the default value, unless the file system you are creating is purely for users (like /home).

Next question: Why are you changing the block and fragment size? Do you expect to write a very large number (many millions) of tiny files? Have you actually thought through the impact this will have on space saving and performance?

Lastly: I always use soft updates, but I'm not sure whether journaling (-j) a gjournal (-J) are considered a good or bad idea; I don't use them (but them, all my production file systems are on ZFS).

 

[Terry Mester]

You can set -m to zero. Consider the consequences if this is the root disk, or any disk holding directories that are necessary for normal system operation (such as /etc, /var, /usr, and so on): ...

This is just a second 'slave' Hard Disk Drive -- like a Floppy Disk, and it would not contain any System Files -- just my personal Files. I don't see why the OS would need Free Space on a Slave Drive, but I want to know for sure.

... Why are you changing the block and fragment size? ... Have you actually thought through the impact this will have on space saving and performance? ...

Wouldn't a smaller Fragment Size save on lost Disk space? At 512 Bytes you never lose more than 511B.

 

[ralphbsz]

If the disk doesn't contain any operating system files, then 0% reserved space is not a problem.

The space-saving argument is interesting. Yes, indeed with a block size of 512 bytes, you will never waste more than 511 bytes on a partial block. On the other hand: what fraction of the disk space is actually used by the "wasted" space at the end of a file? If we assume that file sizes are distributed uniform randomly (warning: that's a nonsensical assumption, they typically have a zip distribution that's partially quantized to the favorite size of the underlying software), the waste is on average 1/2 of a block. For the default 4K block size, that would be on average 2K per file. So if you had a million files on disk, you would be wasting 2GB. That might sound like a lot, but in reality it is very little. To begin with, modern disks are sized in TB, and this is roughly 1/1000th part of it. Second, you probably don't have a million files on disk. I just checked /home on my server machine (which has a 3TB ZFS file system which is roughly 1/3 full), and it has only 1/4 million files. So the waste on my file system is much less. The reason for this is that files are on average much larger than 4K, so the waste is typically a small fraction (the average file size on my file system seems to be roughly 1TB / quarter million, or about 4MB).

Now look at the other side, still from the capacity viewpoint. If you make the block size smaller, you increase the amount of metadata (inodes, indirect blocks, ...) that need to be kept. Because suddenly there is 8x more bookkeeping: every inode (which describes the content of a file on disk) needs to contain 8x more locations of blocks on disk, and it becomes much more likely that the inode overflows the inode size limit (which is typically a block or a sector, I don't know the UFS implementation that well) and an indirect block needs to be allocated to keep track of the data. That will all use space. For big files, this inefficiency by far overwhelms the saving from wasted space. Why? People may think they have a lot of small files, which may even be true, but in reality most of the disk space is used by large files. Making those large files inefficient to store hurts much more, than the little bit of waste at the end of the file helps.

The next argument is performance: keeping trick of 8x more things slows everything down. It eventually, when the file system gets full, also slows disk IO down: imagine a very fragmented file system (you said above you wanted to be willing to get the file system to be very very full, which is why you reduced the reserved space down to 0). Now a 1MB file will have to be written in tiny little blocks. Instead of 250 blocks of 4K each, it will be written in 2000 blocks of 512 bytes each. Since the file system was very fragmented, those 2000 blocks were not contiguous, duh. So now to read the file you'll have to do about 2000 disk seeks. Assuming your disk can do ~100 seeks per second (typical for SATA), reading the file will now take 20s. If in the same situation (totally fragmented, file system nearly full) you had used 4K blocks, it would have taken 2.5s, which is a lot faster ... noticeably so (try counting to 20 for a 1M file, not fun). By the way, if you had kept your file system empty enough so the file could be written contiguously, reading it would have taken about 20ms, not long enough to blink.

I used to work in file system for very large supercomputers. I used to recommend that most customers use a block size of 16MiB (not KiB), and for customers with large files (multiple GB, which is not uncommon in analytics and scientific computing), that they go to 64MiB block sizes. Even if you have an occasional small file (and waste a lot of space), on average you are still coming out WAY ahead. The reason is the simple observation: even if you have a lot of small files, for most users most of the space is in the big files, so those need to be optimized for space efficiency and IO performance. Only users who have nearly only tiny files need to worry about small blocks.

 

[bds]

I think -j (journalled soft-updates) implies -U but -J (geom-level journalling) is a different beast altogether - you'd normally only use that after you've created a gjournal provider underneath. Either remove -J (I think newfs will warn about it and then ignore it) or spend a while reading up on what gjournal is and does.

 

[Chris_H]

To those comments in this thread that are in regards to *NIX/FreeBSD security -- especially FreeBSD.
I would only add;
It's the users responsibility to determine the security policy.
It's the Operating System' responsibility to provide the means.

--Chris

 

[Terry Mester]

... the waste is on average 1/2 of a block. For the default 4K block size, that would be on average 2K per file. So if you had a million files on disk, you would be wasting 2GB. That might sound like a lot, but in reality it is very little. To begin with, modern disks are sized in TB, and this is roughly 1/1000th part of it. ...

Thanks for these details. I save a lot of Webpages ('Webpage Complete' in Windows lingo), and those include many tiny Files in a subdirectory. This is what causes a larger Fragment Size to waste a lot of space. It's an older Drive of 40GB with only 11GB for the FreeBSD Partition, and so wasted space is a legitimate concern. Also, I basically never open the Files I've saved, and so the issue of extra time reading the File for loading is not a concern with long-term storage.

... If you make the block size smaller, you increase the amount of metadata (inodes, indirect blocks, ...) that need to be kept. Because suddenly there is 8x more bookkeeping: every inode (which describes the content of a file on disk) needs to contain 8x more locations of blocks on disk, and it becomes much more likely that the inode overflows the inode size limit (which is typically a block or a sector,...

Do I understand correctly that the UFS Format keeps a Log File of all the Sectors comprising each specific saved File? If so this doesn't seem to be the most efficient way of organizing Disk space. The DOS for my ATARI Computer from thirty years ago contained a simple reference to the "Next Sector Number" in each Sector of a File (with "0" for the Last Sector). The Directory of Files would have only contained the "First Sector Number". This seems to be much more efficient than a Log File of all Sectors comprising every File on the Disk.

I used to work in file system for very large supercomputers. ...

Your knowledge of Disk storage is most interesting. Can you explain how they are able to manage to make a 3-1/2 Inch Hard Disk able to hold so much data as 2 Terabytes? I have a Sony external 2-1/2 Inch HDD which is 1TB, and they also came in 2TB! Twenty-five years ago my first PC Hard Drive only stored 10MB, and the 3-1/2 Inch Floppy Disk stored 1.44MB. The 5-1/4 Inch Diskette only stored 1.2MB. How come they are not able to store such large amounts on the Floppy Disk? Also, aren't these high storage Disks much more vulnerable to electromagnetic frequencies? In order to increase data storage they are decreasing the physical size of each magnetic spot (Bit) on the Disk which makes it more vulnerable to loss.

... Either remove -J (I think newfs will warn about it and then ignore it) or spend a while reading up on what gjournal is and does.

Thanks for catching this! You're right; the "-J" Attribute is not appropriate to my situation.

 

[ralphbsz]

Do I understand correctly that the UFS Format keeps a Log File of all the Sectors comprising each specific saved File? If so this doesn't seem to be the most efficient way of organizing Disk space. The DOS for my ATARI Computer from thirty years ago contained a simple reference to the "Next Sector Number" in each Sector of a File (with "0" for the Last Sector). The Directory of Files would have only contained the "First Sector Number". This seems to be much more efficient than a Log File of all Sectors comprising every File on the Disk.

It's not a "log". The word "log" is a term of art, which means something that gets appended to. Instead the block location metadata is more like a directory or index.

The way it works in MOST Unix file systems (I'm not actually intimately familiar with UFS, and I've never looked at the ZFS source code) is that each "file" is in reality a structure called an inode, which is a file descriptor on disk (remember, in Unix-style systems, zero or more directory entries can point at one file, so the file name in a directory does not have 1:1 correspondence to actual files). The inode contains lots of metadata about the file, like the time stamps, permissions, and so on. Inodes are typically stored on disk in a sector by themselves (so they are often 512byte or 4K long), although some file systems use larger inodes. The remaining space in the inode is the used for some sort of data layout description. Typically this is an index of disk locations where the file is. Typically this is not done sector by sector (that would be too inefficient), but using larger entities (allocation blocks or extents or stripes), or ranges of sectors. If the file is so large that the layout information doesn't fit into the inode, then the file system creates extra blocks (typically called indirect blocks), and then the inode points at indirect blocks, which then contain the same data structure describing where the data is. For really large files, there may be even multiple layers of indirect blocks. Some file systems use an interesting optimization: If the file is very small, instead of storing layout information in the inode, the actual file content gets stored there (depending on inode size, this works for files up to a few hundred bytes, and on file systems with 4K inodes up to several K). This is a great optimization for file system with lots of tiny files. I don't think UFS uses the optimization; in ZFS, the concept of "inode" is more complex, and I don't know whether such an optimization even applies.

The technique you describe from Atari would be a really bad idea with todays file systems. Two reasons: First, the "cargo" content of each sector would be less than 512 bytes (some space has to be reserved for the pointers). That means copying files from disk to memory would be very hard: instead of letting a DMA engine in the disk controller just copy large blocks of data (many sectors, sometimes with IO sizes of many MB) directly from disk, it would have be first copy into a holding buffer, and then copied back into a separate memory buffer while removing the space reserved for the link pointers. That double copy would be performance killing. Second: How to you do random access to the middle of a file? In today's inode-based file systems, you just need to read the inode, perhaps a chain of a few indirect blocks, and you can go directly to any piece of the file in the middle. With the Atari technique, you would have to run sequentially through the whole file, and for large files (I've seen files that are multiple TB, and files that are multiple GB are common today) that would be very slow.

Can you explain how they are able to manage to make a 3-1/2 Inch Hard Disk able to hold so much data as 2 Terabytes?

Honestly, only in general terms. I'm a software person, not a hardware person.

To begin with, the limit today I think is 14 or 16 TB in shipping products; the largest I've held in my own hands is 10TB. How does this work? The physicists who work for Seagate and WD=Hitachi are very very smart, hardworking, and friendly. Actually, some of them are my friends, and I occasionally go drink with them (given my age, mostly with retired ones). Over the years, the size of each bit on disk has become smaller, both as the tracks that the bits sit on have gotten closer to each other, and the density of bits along the track has increased. Today they use truly bizarre magnetic materials on the platter (it hasn't been iron oxide for ages, today's platters aren't brown, their silver), very strange magnetic semiconductors in the read/write heads, they orient the bits perpendicular to the platter to make it smaller (that's called PMR), they partially erase neighboring bits while writing new ones (called SMR), they fry the media or shine lasers on them while writing (that's HAMR and friends), they fly the heads laughably low (so low that the heads way too often touch the thin coating of lubricant applied to the surface, which today is actually causing wear limits on disk drives), and they are generally very careful about quality control and repeatability. These guys are really good at what they do. Amazingly enough, they still manage to make drives that cost about $300 each.

One drawback of this is: Because of electrical limitations in the heads (the data frequency can't go much over several Gigahertz), the highest capacity drives now have to spin more slowly. That's why the 10K and 15K RPM drives no longer have really high capacity. Old joke: Good, fast, cheap, pick any two.

Also, aren't these high storage Disks much more vulnerable to electromagnetic frequencies?

Not that I have heard of. For a while, disk drives were too vulnerable to vibration, which was particularly a problem if you have an enclosure with many disks mounted near each other. Both the people who make disks and the ones who make enclosures have learned from their mistakes. But magnetic fields or EM noise have not been a particular problem, at least as far as I've seen.

 

[Terry Mester]

It's not a "log". The word "log" is a term of art, which means something that gets appended to. Instead the block location metadata is more like a directory or index. ...

Thanks again Ralph for all of this very interesting information. Upon reflection, I suppose that the DOS method of keeping a 'Directory' of Disk Sectors per File would be faster for saving Files to Disk than the older ATARI method of recording the next Sector in each Sector. The ATARI method would require reading every Sector to find the empty ones every time the DOS needs to save a File, and this would of course be slower.

... very strange magnetic semiconductors in the read/write heads, they orient the bits perpendicular to the platter to make it smaller (that's called PMR), they partially erase neighboring bits while writing new ones (called SMR), they fry the media or shine lasers on them while writing (that's HAMR and friends), they fly the heads laughably low (so low that the heads way too often touch the thin coating of lubricant applied to the surface, which today is actually causing wear limits on disk drives), ...

Some of this concerns me as to how reliable such a Hard Disk would be. I would never want a Hard Drive which cannot be trusted to not lose my data. I personally would prefer reliability to high capacity.


Back to the original issue; I formatted the FreeBSD Partition on my Hard Drive 1 as follows. It successfully mounts from the 'fstab' File.
newfs -U -b 4096 -f 512 -j -m 0% /dev/ada1s2a

'newfs' provided the following output message:
Warning: changing optimization to space because minfree is less than 8%
/dev/ada1s2a: 8466.2MB (17338817 sectors), block size 4096, fragment size 512
using 914 cylinder groups of 9.27MB, 2372 blks, 9488 inodes.
with soft updates
Super-block backups (for fsck_ffs -b #) at:
144, 19120, 38096, 57072, ... etc.
Using inode 4 in cg 0 for 9715712 byte journal
newfs: Journal file fragmented
newfs: soft updates journaling set

Is there anything to point out about this message?
Also, gpart show reveals that the first 63 Blocks (Sectors) of each Hard Drive are unassigned. Why would this be?

 

[ralphbsz]

The unassigned 63 blocks contain the GPT partition table and similar stuff. The size of that area ends up being determined by aligning the first "real" partition correctly.

I still don't like your small block and fragment size. If you get reasonable performance with it, more power to you. If things seem slow, then try the defaults.

You wrote: "I would never want a Hard Drive which cannot be trusted to not lose my data. I personally would prefer reliability to high capacity." Sorry, the reality is that no hard drive is terribly reliable. The MTBF (mean time before failure) specification for modern enterprise hard drives is about 1.5 million hours or about 150 years; the reality is that disks under ideal conditions (data centers, good enclosures, reasonable and constant temperatures) live half that long. For consumer disks, it's even less. That means that every year the probability that your drive completely fails is several percent. The probability of read errors without disk failure is even much higher. Disk drives are simply not reliable, at the scale of looking for decades of storage, or if you have multiple drives (if you had 100 drives, you'd have to replace a few drives every year). If you want serious reliability (likelihood of data loss way below 1% per year), you NEED TO use redundancy of some form. There are several ways to do that. One is do to very regular backups (every day?) to a second and third disk drive, or to the cloud. The problem with backups is that they leave a small window of vulnerability, and if your disk fails, you have to do a lot of work to restore them. Personally, I recommend RAID: Get two (or more) disk drives, and use some software (I like ZFS) to gang them together. The probability that both disks fail at the same time (or have an error in the same place) is low. For professional enterprise use you need at least 3 disks (because the probability that the surviving disk has a read error when the first disk fails is high enough to be considered unsafe, for the TB quantities of data and high reliability needs of the enterprise), but for amateur home use, a pair of disks is usually good enough.