A web-interface to ZFS (call for testers)

S

sub_mesa

Hello all BSD lovers!

Update: 0.1.7-preview3 release available!

As i love FreeBSD, and love ZFS; i run those two at home. However, many people still store their files using shitty Windows "FakeRAID" which caused many people to lose data. ZFS would be great for them, but running OpenSolaris or FreeBSD is too big a step for someone who does not know what a shell is.

For those users, FreeNAS has been an excellent BSD-derived product that implements a nice web-interface, and also supports experimental ZFS in recent releases. But still not FreeBSD 8 and not the newer ZFS versions (13/14) available now; which fixes many bugs in the process. Also, FreeNAS has a stripped down FreeBSD userland.

For those reasons, i decided to write a web-interface to ZFS that runs on FreeBSD 8 or later (even 9). I wanted to create something that adds to FreeBSD, not the other way around. Thus, you would have a full working FreeBSD system with a web-interface; both should be able to update and use interdependently. Plus, you can start to learn FreeBSD's other advanced and powerful features while getting core functionality of a NAS working quickly using the web-interface.

Currently, i only have a script-tarball test (pre-)release. I plan on something that runs in Virtualbox or can be put on a 1GB pendrive, as well.

Here's a screenshot to keep things in perspective:

mesa-sneakpreview16.png

(screenshot updated)

It's current features are:
  • configuring network interfaces
  • enabling/disabling DHCP
  • formatting disks
  • creating disk labels
  • basic disk benchmarking
  • create and destroy zpools
  • display importable zpools (does not actually import yet)
  • expanding zpools
  • scrub zpools
  • create and destroy filesystems
  • command line (including root access)
  • shutdown or reboot
  • upgrade via HTTP file upload (import)
  • upgrade via remote file download (import)
  • export and upload current scripts via HTTP (export)

I invite anyone to try and test my scripts; i'm looking for feedback at this point. Most core functionality is implemented. Sharing of files via Samba/NFS still is not; this will be addressed in a future release.

WARNING: Please, do not test this on systems with real data. It should be considered alpha-quality software intended for testing only. You can do that in Virtualbox; just make some virtual disks and you can test the various ZFS functions in the interface. Again: DO NOT RUN on systems with real data!

Install instructions:
1. Install FreeBSD using my FreeBSD install guide
2. Setup FreeBSD using my FreeBSD setup guide
3. Configure FreeBSD as webserver and install my scripts, using my Mesa install guide
4. Now visit the website typing http://<ip-address> in your webbrowser

Feel free to ask any questions. :)
 
Just an aside...

Lots of good zfs threads (posts) in the
subforum (on another forum) which
discusses this GUI in more detail,
possibly with more screenshots of it.
 
sub_mesa said:
For those users, FreeNAS has been an excellent BSD-derived product that implements a nice web-interface, and also supports experimental ZFS in recent releases. But still not FreeBSD 8 and not the newer ZFS versions (13/14) available now; which fixes many bugs in the process. Also, FreeNAS has a stripped down FreeBSD userland.
Click to expand...

Its true, that original FreeNAS developers switched to Debian as a base, but they also renamed their project to CoreNAS, FreeNAS at the same time has been adopted by the iXsystems (PC-BSD owner), so next FreeNAS release may be under development.

You may also create a separate FreeBSD/NAS project to make it your own way, especially that You already have a lot and You can use anything from old FreeNAS project.

I would also consider more 'advanced' install process like that one:
http://forums.freebsd.org/showthread.php?t=12082

You may also want to switch to some 'better' language, like PYTHON (or DJango framework).
 
I now created a raw disk image with everything preinstalled, which can be written directly to USB pendrive or small HDD. It requires 614MB of space once extracted; compressed download-size is about 170MB.

For those interested in trying this out, all instructions are on this page:
http://submesa.com/mesa

But really, all you guys already familiar with FreeBSD need to do is download the tarball, extract it, and then write the .img file using dd:
dd if=mesa-0.1.0-prerelease.img of=/dev/ad1 bs=1m
(example; replace /dev/ad1 with target device to be formatted/overwritten)

Then just boot your system with the target device as primary boot device set in the System BIOS. Once booted it should assign IP by DHCP and have the webserver with ZFS interface up and running (without authentication). Anyone who wants to try this?
 
Just released 0.1.5 LiveCD and Binary (USB/CompactFlash) image release. Both are now based on tmpfs, meaning you need at least 1GB RAM to use this effectively, as 512MB will be reserved by the system.

The LiveCD forgets any Samba or other settings upon reboot. The USB binary image does save a few settings across reboots, like Samba and web-interface preferences file and zfs' zpool.cache which it needs to automount your disks.

A lot of changes under the hood; anyone who could try and test it? Not suitable for real data yet, by the way!

LiveCD: http://submesa.com/files/Mesa-0.1.5-livecd.iso
Binary: follow instructions at http://submesa.com/mesa
 
Update: new version 0.1.7-preview2 released!

LiveCD URL: http:// zfsguru.com/files/ZFSguru-0.1.7-preview2.iso
LiveCD MD5: 7d2774bf2771e7c7eb42bec85527e2d5

This also adds iSCSI functionality you've asked for, interfasys, hope it works well for you. Could anyone test this? If you have a lot of disks with no data on it, you can also use the new benchmarking feature, which creates nice visual graphs in all kinds of different ZFS configurations.


Full changelog so far:

Code: 
Version 0.1.7 (not yet released)

    (System) ZFS system version updated from V14 to V15.
    (System) now includes ZFS metaslab patch, increasing performance on filled pools.
    (System) now loads the Silicon Image controller driver "siis" by default.
    (System) now properly detects Marvell controllers such as SuperMicro AOC-SAT2-MV8.
    (Web) implemented Disks->SMART page, allowing easy SMART monitoring of your disks.
    (Web) implemented Disks->Benchmark page, allowing advanced benchmarking of your disks.
    (Web) now allows creating pools with a lower pool version (spa) or filesystem version (zpl).
    (Web) implemented Files->ZVOLs page, allows creating/destroying ZVOLs.
    (Web) Services page rewritten to allow stopping/starting services manually and automatically upon boot.
    (Web) implemented Services->iSCSI page, allowing iSCSI configurations.
    (Web) implemented Services->OpenSSH page, allowing easy SSH access.
    (Web) implemented System->Tuning page, allows tuning of /boot/loader.conf variables.
    (Web) many more smaller changes to web-interface! 

Version 0.1.6 (September 23rd 2010)

    Project got renamed to ZFSguru.
    Project got a new website at zfsguru.com.
    (All) ability to install to ZFS-on-root; allowing you to boot off your software RAID pool.
    (All) the PHP property 'short_open_tag' is no longer required and default php.ini now disables this feature.
    (System) updated pre-installed packages to newer versions and switched to port-built packages instead.
    (Binary) added 3 second sleep before USB detection to avoid misdetections.
    (Web) new disk detection detects more controllers with easy future extensions possible.
    (Web) formatting new disks redesigned; now allows for both GPT partitions and GEOM labels.
    (Web) now also allows zero-write, random-write and secure-erase when formatting disks.
    (Web) pools page now allows user to import a pool with the click of a button.
    (Web) lots of smaller changes.

Version 0.1.5 (August 11th 2010)

    created using new build system (faster development time).
    (All) fixes Samba/NFS permission issue when creating new pool or filesystem.
    (Binary) now based on FreeBSD 8.1-RELEASE instead of 8.1-RC1.
    (Binary) now uses new boot method (GPT boot) to avoid mountroot boot problems.
    (Binary) now based on the same tmpfs architecture as the LiveCD.
    (Binary) added mod script framework to allow adding packages to the system that survive a reboot.
    (LiveCD/Binary) added checksumming of system disk image to guard against corruption.
    (LiveCD) tmpfs stability fixes (fixes out of space with enough RAM by adding more inodes).
    (Web) properly creates configuration file upon first webpage access.

Version 0.1.4 (July 30st 2010)

    first release that features LiveCD! (still experimental).
    (LiveCD) now based on FreeBSD 8.1-RELEASE instead of 8.1-RC1.
    (LiveCD) has out-of-the-box working Samba configuration.
    (Binary) fixes some permissions issues.
    (Web) fixes disk formatting showing blank page.
    no fix for zpool permissions yet.

Version 0.1.3 (July 23rd 2010)

    (Web) functionality of Files page greatly extended.
    (Web) Samba and NFS sharing now supported.
    (Web) some work on the Pools page; in the future will allow for replacing failed disks.
    (Web) added changelog to the Extra->Help page.

Version 0.1.2 (July 10th 2010)

    (Script) fixed /dev/da (SCSI/RAID) devices not showing up on the Disks page.

Version 0.1.1 (June 21st 2010)

    (Binary) image now based on FreeBSD 8.1-RC1 instead of 8.0-RELEASE.
    (Binary) image now has updated versions of samba and other packages.
    (Web) small additions and fixes.

Version 0.1.0 (June 6th 2010)

    initial release.
Please test and let me know if you see anything that can be improved. It's still lacking some core features like replacing failed disks and email notification, though. But the project is well underway towards its first stable release: 0.2.0.

Cheers!
 
New Release: 0.1.7-preview3

My Christmas present to you guys: preview3 release!
  • ZFSguru 0.1.7-preview3 LiveCD (.iso) containing experimental ZFS v28 system image
  • ZFSguru 0.1.7-preview3 web-interface available for web-update
  • FreeBSD 9.0-001 amd64 system image containing experimental ZFS v28 code
Warning: the ZFS v28 system image, including the one included on the LiveCD, is considered very experimental, and you should not use this on your existing valuable data unless you got a very good backup! This release is for testing only!


System version 9.0-001 amd64 (bleeding edge)
  • based on FreeBSD 9.0-CURRENT 20101221 code + ZFS v28 patch + head patches
  • highly experimental 'bleeding edge' release; DO NOT USE ON REAL DATA!
  • it actually does work, however! Imports your existing pools fine.
  • supports ZFS v28 including de-duplication and RAID-Z3
  • booting support in Virtualbox proved difficult; Virtualbox does not yield access to other devices than the boot device. Please test more thoroughly in the real world! Create a RAID-Z2 and boot from it, have multiple pools of different ZFS versions, etcetera.
  • has the new MPT Fusion 2 'mps' driver supporting LSI SAS2008 and other 6Gbps-family chips. This adds support for SuperMicro USAS2 6Gbps HBAs.
  • has a few fixes for disk labeling (no more gptid labels)
  • has a new menu instead of login prompt, with very basic functionality; but it does display the IP address the web-interface is running on!
  • updated all packages, including Samba to 3.5.6 (up from 3.4.x); this may have performance implications.

ZFSguru web-interface version 0.1.7-preview3
  • fixed issues with ZFSv28 new data format
  • adds De-duplication support to Files page, on pools and systems that support it.
  • changed automatic memory tuning to new defaults (kmem=1.5x the RAM)
  • fixed automatic memory tuning on Root-on-ZFS installations
  • some important of under-the-hood changes

How to upgrade?
  • first update your Web-interface on the System->Update page, to preview3 version
  • next, System->Install page will allow you to perform Root-on-ZFS installation and download the newly available 9.0-001 system image to your system and use it for installation.
  • after installation reboot and you should run the new experimental ZFS v28 system!

Note that in order to use de-duplication, you must either create a pool of at least pool version 21, or upgrade an existing pool with the zpool upgrade command. Note that if you do this, you cannot go back to stable ZFS versions like the ZFS v15 the stable ZFSguru versions use.

So please test, but be careful not to burn your fingers! Bleeding edge can bite you. :)

If there are no critical issues with this release, it will be branched 0.1.7 (final) and next release will be 0.1.8 in new year. The 0.1.7 final will contain the 'stable' ZFS v15 system image based on FreeBSD 8.2-RC; so this really is a testing-only release.

Cheers!
 
You must log in or register to reply here.
Back
Top