GlusterFS

Latest revision as of 19:15, 28 February 2011

[edit] GlusterFS

GlusterFS is a free software parallel distributed file system, capable of scaling to several petabytes.

GlusterFS is a network/cluster filesystem. GlusterFS package comes with two components, a server and a client. The storage server (or each in a cluster) runs glusterfsd and the clients use mount command or glusterfs client to mount the exported filesystem. Storage can be kept scaling beyond petabytes as demand increases.

To mount GlusterFS file systems, the client computers need FUSE support in the kernel.

For details, see http://en.wikipedia.org/wiki/GlusterFS

Installing

At the time of this writing, there is no FreeBSD port for gluster, you will first need to install FUSE manually.

cd /usr/ports/sysutils/fusefs-kmod
make install clean
echo "fusefs_enable="YES"">>/etc/rc.conf
sh /usr/local/etc/rc.d/fusefs start

Next you will need to compile and install the glusterFS manually.

cd /tmp
fetch http://ftp.zresearch.com/pub/gluster/glusterfs/1.4-pre/glusterfs-1.4.0pre5.tar.gz
tar -zxf glusterfs-1.4.0pre5.tar.gz
cd glusterfs-1.4.0pre5

export LDFLAGS="-L/usr/local/lib -liconv"
export CFLAGS="-O0 -g -DDEBUG -I/usr/local/include/"
./configure -enable-fuse-client
make install clean

Freebsd 7: Both client and server were installed successfully

Known Issues

ISSUE #1 (Freebsd 7 client) - cannot use fstab to mount the fuse partition:

glusterfs -f /usr/local/etc/glusterfs/client.vol /usr/local/www/mantis/files  => works
 
FSTAB:   /usr/local/etc/glusterfs/client.vol        /usr/local/www/mantis/files     glusterfs    rw    0     0
mount: /usr/local/etc/glusterfs/client.vol : Operation not supported by device => the error

ISSUE #2 (Freebsd 6.2 client) – Client segfault

[root@martini] $ glusterfs --no-daemon --log-file=/dev/stdout --log-level=DEBUG -f /usr/local/etc/glusterfs/client.vol /usr/local/www/mantis/files
2008-10-03 16:33:40 D [glusterfs.c:271:_get_specfp] glusterfs: loading volume specfile /usr/local/etc/glusterfs/client.vol
 
Version      : glusterfs 1.4.0pre5 built on Oct  2 2008 21:47:51
TLA Revision : glusterfs--mainline--3.0--patch-359
Starting Time: 2008-10-03 16:33:40
Command line : glusterfs --no-daemon --log-file=/dev/stdout --log-level=DEBUG -f /usr/local/etc/glusterfs/client.vol /usr/local/www/mantis/files 
given volume specfile
+-----
  1: ##############################################
  2: ###  GlusterFS Client Volume Specification  ##
  3: ##############################################
  4: 
  5: volume gohabs
  6:   type protocol/client
  7:   option transport-type tcp/client
  8:   option remote-host gohabs 
  9:   option remote-subvolume mantis
 10: end-volume
 11: 
 12: volume bondeau
 13:   type protocol/client
 14:   option transport-type tcp/client
 15:   option remote-host bondeau
 16:   option remote-subvolume mantis
 17: end-volume
 18: 
 19: volume afr
 20:    type cluster/afr
 21:    subvolumes bondeau gohabs
 22: end-volume
+-----
2008-10-03 16:33:40 D [spec.y:194:new_section] parser: New node for 'gohabs'
2008-10-03 16:33:40 D [xlator.c:289:xlator_set_type] xlator: attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/protocol/client.so
2008-10-03 16:33:40 D [spec.y:219:section_type] parser: Type:gohabs:protocol/client
2008-10-03 16:33:40 D [spec.y:285:section_option] parser: Option:gohabs:transport-type:tcp/client
2008-10-03 16:33:40 D [spec.y:285:section_option] parser: Option:gohabs:remote-host:gohabs
2008-10-03 16:33:40 D [spec.y:285:section_option] parser: Option:gohabs:remote-subvolume:mantis
2008-10-03 16:33:40 D [spec.y:367:section_end] parser: end:gohabs
2008-10-03 16:33:40 D [spec.y:194:new_section] parser: New node for 'bondeau'
2008-10-03 16:33:40 D [xlator.c:289:xlator_set_type] xlator: attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/protocol/client.so
2008-10-03 16:33:40 D [spec.y:219:section_type] parser: Type:bondeau:protocol/client
2008-10-03 16:33:40 D [spec.y:285:section_option] parser: Option:bondeau:transport-type:tcp/client
2008-10-03 16:33:40 D [spec.y:285:section_option] parser: Option:bondeau:remote-host:bondeau
2008-10-03 16:33:40 D [spec.y:285:section_option] parser: Option:bondeau:remote-subvolume:mantis
2008-10-03 16:33:40 D [spec.y:367:section_end] parser: end:bondeau
2008-10-03 16:33:40 D [spec.y:194:new_section] parser: New node for 'afr'
2008-10-03 16:33:40 D [xlator.c:289:xlator_set_type] xlator: attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/cluster/afr.so
2008-10-03 16:33:40 D [xlator.c:324:xlator_set_type] afr: strict option validation is not enforced -- neglecting
2008-10-03 16:33:40 D [spec.y:219:section_type] parser: Type:afr:cluster/afr
2008-10-03 16:33:40 D [spec.y:352:section_sub] parser: child:afr->bondeau
2008-10-03 16:33:40 D [spec.y:352:section_sub] parser: child:afr->gohabs
2008-10-03 16:33:40 D [spec.y:367:section_end] parser: end:afr
2008-10-03 16:33:40 D [xlator.c:289:xlator_set_type] xlator: attempt to load file /usr/local/lib/glusterfs/1.4.0pre5/xlator/mount/fuse.so
2008-10-03 16:33:40 D [xlator.c:324:xlator_set_type] fuse: strict option validation is not enforced -- neglecting
2008-10-03 16:33:40 D [glusterfs.c:771:main] glusterfs: running in pid 99336
 
2008-10-03 16:33:40 D [fuse-options.c:140:fuse_options_validate] fuse-options: using mount-point = /usr/local/www/mantis/files
2008-10-03 16:33:40 D [fuse-options.c:147:fuse_options_validate] fuse-options: using attr-timeout = 1
2008-10-03 16:33:40 D [fuse-options.c:159:fuse_options_validate] fuse-options: using entry-timeout = 1
2008-10-03 16:33:40 D [fuse-options.c:171:fuse_options_validate] fuse-options: using direct-io-mode = 1
2008-10-03 16:33:40 W [common-utils.c:158:gf_print_bytes] glusterfs: Total data (in bytes): transfered (0), received (0)
pending frames:
 
Signal received: 11
configuration details:db.h 1
dlfcn 1
libpthread 1
spinlock 1
extattr.h 1
package-string: glusterfs 1.4.0pre5
Segmentation fault: 11 (core dumped)
 
[root@martini] $ gdb glusterfs glusterfs.core
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd"...
Core was generated by `glusterfs'.
Program terminated with signal 11, Segmentation fault.
Reading symbols from /usr/local/lib/libglusterfs.so.0...done.
Loaded symbols for /usr/local/lib/libglusterfs.so.0
Reading symbols from /usr/local/lib/libiconv.so.3...done.
Loaded symbols for /usr/local/lib/libiconv.so.3
Reading symbols from /lib/libpthread.so.2...done.
Loaded symbols for /lib/libpthread.so.2
Reading symbols from /lib/libc.so.6...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /usr/local/lib/glusterfs/1.4.0pre5/xlator/protocol/client.so...done.
Loaded symbols for /usr/local/lib/glusterfs/1.4.0pre5/xlator/protocol/client.so
Reading symbols from /usr/local/lib/glusterfs/1.4.0pre5/xlator/cluster/afr.so...done.
Loaded symbols for /usr/local/lib/glusterfs/1.4.0pre5/xlator/cluster/afr.so
Reading symbols from /usr/local/lib/glusterfs/1.4.0pre5/xlator/mount/fuse.so...done.
Loaded symbols for /usr/local/lib/glusterfs/1.4.0pre5/xlator/mount/fuse.so
Reading symbols from /usr/local/lib/libfuse.so.2...done.
Loaded symbols for /usr/local/lib/libfuse.so.2
Reading symbols from /libexec/ld-elf.so.1...done.
Loaded symbols for /libexec/ld-elf.so.1
#0  0x281cb537 in pthread_testcancel () from /lib/libpthread.so.2
[New Thread 0x805e400 (sleeping)]
[New Thread 0x805e200 (LWP 100102)]
[New Thread 0x805e000 (LWP 100143)]
(gdb) bt
#0  0x281cb537 in pthread_testcancel () from /lib/libpthread.so.2
#1  0x281b9ee5 in sigaction () from /lib/libpthread.so.2
#2  0x281c2d31 in pthread_mutexattr_init () from /lib/libpthread.so.2
#3  0x281c2d9f in pthread_mutexattr_init () from /lib/libpthread.so.2
#4  0x2827a6bf in _ctx_start () from /lib/libc.so.6
#5  0x00000000 in ?? ()
#6  0xbfbfe460 in ?? ()
#7  0xbfbfe1a0 in ?? ()
#8  0x00000000 in ?? ()
#9  0x281c2d5c in pthread_mutexattr_init () from /lib/libpthread.so.2
#10 0x282f035b in init (this_xl=0x8056880) at fuse-bridge.c:2694
#11 0x0804ba34 in main (argc=7, argv=0xbfbfebb0) at glusterfs.c:775
(gdb) quit
[root@martini] $ uname -imsr
FreeBSD 6.2-STABLE i386 GENERIC