History of ZFS
Part 1: The Birth of ZFS and How it All Started
This is part of our article series published as “History of OpenZFS”. Subscribe to our article series to find out more about the secrets of OpenZFS
ZFS (or the Zettabyte File System) is approaching its 15th birthday, and over a decade since integration into FreeBSD. Originally created by Sun Microsystems, ZFS grew in popularity because of its advanced features. Today we will take a look at its history.
A Problem in Search of a Solution
ZFS came about “because of a typo”. Back in the late 1990s, Sun Microsystems had a system named Jurassic that they used to host the home directories of their employees. At the time, Jurassic was running the Unix File System or UFS as the file system and the Solaris Volume Manager to handle management of the disks. While performing routine maintenance, an engineer named Mike Sullivan mistyped a command for the SVM.
A simple mistake caused a huge fallout. “Everything came crashing down because the device mapping was now wrong. So, UFS was seeing wrong block from the file system.” “SVM, like a lot of these software-based volume management tools, they had their own layering, they had they own idea of labels. If it got confused about which devices were part of the RAID group and which weren’t, then everything was all over. You might have your data there, but if the label wasn’t there and it couldn’t figure it out, you date was essentially all gone.” The net result was that a thousand engineers were left with nothing to do while the system was restored from backup.
While the backup was being restored, Jeff Bonwick went to talk to Tim Marslind, another Sun engineer. They started to complain about Sun’s inability to create a new file system after several failed starts. Sun had started several projects to create a new file system, but ultimately shut down the projects after several months.
They also pondered why storage management wasn’t easier. After all, it should be just like adding memory to a system, just add it and reboot. The system just starts using the added memory and everything works faster. “[There’s no DIMM config](
) that you have to run. You don’t create virtual DIMMs. There’s no DIMM management software…Why can’t you just treat your disk as a pool of storage and you allocated from them with a memory allocator…That became the SPA, the Storage Pool Allocator…Just like in VM system, where you have an MMU (Memory Management Unit) to manage translations of things from virtual to physical. We could have a Data Management System.” Unfortunately, nothing came out of this conversation at the time. Both engineers went back to their respective projects after their data was restored to Jurassic.
A Failed Start
Around the year 2000, Bonwick decided that he wanted to take a crack at creating a new file system after yet another file system project failed. He went to Mark Himmelstein, director of the Operating System Group. Bonwick told Himmelstein that he wanted a small team of “five or six people in the same building” to work on a new file system. Himmelstein said that he needed someone to lead a team of 80 engineers named the Data Organization. He told Bonwick that “I need a data architect, so if you take that role, you can go do the file system.”
Bonwick called what followed “the worse year of my career”. He could not get the members of the team to “buy into” his project. The team was divided between Colorado, Los Angeles, and Santa Clara. Bonwick spent quite a bit of time flying between the locations and was unable to get any momentum going and the project crashed.
Getting Down to Business
In the fall of 2000, Bonwick’s third son was born. At the time, he took six weeks off. He used that time to collect his thoughts. When he returned to Sun, he decided that he would take one last stab at creating a new file system with only one other person, a new hire named Matthew Ahrens. They started working July 20, 2001, and by Halloween, they had a working prototype. Ahrens had worked on creating the data management unit and Bonwick wrote the storage pool allocator. Since they had a working prototype, Himmelstein gave them permission to continue work on their new file system.
Did you know?
You can maximize the power of your ZFS infrastructure with our Support Subscription!
Check it out!They had several things that they wanted to focus on. According to Bill Moore, “one of the design principles we set for ZFS was: never, ever trust the underlying hardware”. They also wanted to simplify data management. Finally, they wanted to make sure the file system had good performance. “So, unless you make it fast, people will fundamentally be uninterested—except for those who have experienced data corruption firsthand.”
The Team Grows
With the blessing of the higherups, they brought on more engineers, including Mark Maybee and Mark Shellenbaum. Eventually the team grew to over twelve people. Maybee and Shellenbaum were brought into the project because they were “file systems people, who knew how to write a file system from scratch“. In fact, Maybee and Shellenbaum joined Sun in the late 1990s “with a carrot dangled in front of us that we were going to possibly get to write a new file system.” That hadn’t happened until the ZFS project started.
So, the two Marks worked in Colorado and Bonwick, and Ahrens worked in California. Maybee and Shellenbaum worked on creating the ZFS POSIX Layer. This allowed the operating system to talk to the file system. On October 31, 2002, they had their first kernel mount. This was a big step towards their first stable release.
As their project started to come together, the ZFS team started using the new file system to store their own files in 2004. Finally, on October 31, 2005, they integrated ZFS into Solaris, Sun’s Unix-based operating system. After this milestone, ZFS went from being using by about a dozen people to over a thousand.
In 2006, Sun Microsystems officially released ZFS to the public as part of their Solaris 10 operating system. The original code for ZFS was proprietary. However, Sun was a big supporter of open-source ideals. So, it wasn’t long before they open-sourced most of ZFS’ code. But that part of the story is for another day. Stay tuned!
Like this article? Share it!
- Share on Facebook
- Share on Twitter
- Share on Linkedin
- Share via Email
- Share on Reddit
You might also want be interested in
Improve the way you make use of ZFS in your company
ZFS is crucial to many companies. We guide companies and teams towards safe, whitepaper implementations of ZFS that enhance and improve the way the infrastructure is enabling your business.
ZFS Support >
ZFS Development >
Continue reading...