how to start a baby AFS cell (openafs, as built for umce). Pre-setup: is time right? date ntpq xntpdc ps do you have DNS setup right? host dig Is hostname right? (ptserver at least cares) hostname cache manager and kernel extension should be INSTALLED or loaded in the kernel. Cache manager (and "afsd") need NOT be running (but to get a token must run afsd, to make cell & files must mount /afs). lsmod ps To configure a kerberos & pt only AFS cell (NO file service) the following files and directories are required: umce /usr/afs/bin/ bosserver /tmp/ <- bad home... asetkey /usr/afs/bin/ ptserver /usr/local/bin/ pts /usr/afs/local/ sysid /usr/afs/logs/ /usr/afs/db/ /usr/afs/etc/ CellServDB KeyFile ThisCell UserList /usr/afs/local/ BosConfig (bad home, but for now, get asetkey from /afs/umich.edu/group/itd/build/mdw/umce/tmp/asetkey ) umce /etc krb5.conf /usr/krb5/var/krb5kdc/ kdc.conf kadm5.acl .k5.$MY_NEW_REALM kadm5.keytab /usr/krb5/sbin/ kdb5_util krb5kdc kadmind krb524d kadmin kadmin.local ktutil fakeka /etc/services setup. Make sure these entries exist with these values. This MUST be done for replication. It otherwise avoids some hassles and warnings. # # added for the UMICH.EDU cell # krb5_prop 754/tcp krb5_rep 755/tcp kpop 1109/tcp #Kerberos POP server krb524 4444/tcp # Kerberos 5 to 4 ticket translator uniqname 48138/udp # Unique Login Name program aserver 48139/udp # Account Server kauthd 48145/udp Kerberos configuration: edit krb5.conf In [libdefaults] default_realm = MY_NEW_REALM # replace any existing. if [libdefaults] if default_tgs_enctypes default_tkt_enctypes permitted_enctypes are specified, make sure des-cbc-crc is included. In [realms] add this, MY_NEW_REALM = { kdc = admin_server = } If [logging] is missing, [logging] kdc = FILE:/var/log/krb5kdc.log admin_server = FILE:/var/log/kadmin.log default = FILE:/var/log/krb5lib.log Rplace "MY_NEW_REALM" with desired realm name; must be upper-case. Replace with FQDN of server. (edit filenames as needed for local convention...) Edit /usr/krb5/var/krb5kdc/kadm5.acl make sure there's a line that reads: */admin@MY_NEW_REALM * admin@MY_NEW_REALM * krb5.conf must be present on each client machine that can access this cell. kdc.conf must be present on any machine which runs kadmin or kadmind. kadm5.acl need only be present on kdc. Make sure /usr/krb5/var/krb5kdc/kdc.conf exists and has appropriate contents. Sample entry (change hostname and realm as needed): [libdefaults] kdc_ports = 88,750 v4_mode = nopreauth [realms] MY_NEW_REALM = { database_name = /usr/krb5/var/krb5kdc/principal admin_keytab = /usr/krb5/var/krb5kdc/kadm5.keytab acl_file = /usr/krb5/var/krb5kdc/kadm5.acl key_stash_file = /usr/krb5/var/krb5kdc/.k5.MY_NEW_REALM kadmind_port = 749 max_life = 100h 0m 0s max_renewable_life = 7d 0h 0m 0s master_key_type = des-cbc-crc supported_enctypes = des3-hmac-sha1:normal \ des-cbc-crc:normal des-cbc-crc:afs3 kdc_supported_enctypes = des3-hmac-sha1:normal \ des-cbc-crc:normal des-cbc-crc:afs3 } [logging] kdc = FILE:/usr/krb5/logs/krb5kdc admin_server = FILE:/usr/krb5/logs/kadmin default = FILE:/usr/krb5/logs/krb5lib.log Beware non-standard "key_stash_file" above. Your K5 binaries look at the wrong files? Check out these env vars for relocating things at run-time (shown with their defaults if you don't specify these): KRB5_CONFIG=/etc/krb5.conf KRB5_KDC_PROFILE=/usr/krb5/var/krb5kdc/kdc.conf Kerberos database initialization /usr/krb5/sbin/kdb5_util create -s will ask for a master key. Used mainly to encrypt db. Can be used for "split secrets" backup of kerberos database. Make some required principals: ( \ below means line broken for clarity; DON'T TYPE \ newline.) /usr/krb5/sbin/kadmin.local kadmin.local: ank admin kadmin.local: ktadd -k /usr/krb5/var/krb5kdc/kadm5.keytab \ kadmin/admin kadmin/changepw kadmin.local: quit Refer to MIT kerberos-5 "installation guide" if you want to enable MIT K5 replication. Start kerberos using: stand-alone (for now): /usr/krb5/sbin/kadmind /usr/krb5/sbin/krb5kdc /usr/krb5/sbin/krb524d -m Now, kinit admin klist kdestroy should work. CellServDB must contain entries for the new AFS cell, plus also entries for any other cells that anything (especially unserver) might wish to contact. So, /usr/afs/etc/CellServDB >my.cellname #what is it? my.ip.addr.here.x #my.dns-name.here /usr/afs/etc/ThisCell my.cellname may need to be created. On the client side, /etc/openafs/CellServDB should have the contents of the server file appended. Don't touch /etc/openafs/ThisCell unless it already points to an existing cell with a valid root.afs . If you don't have any writeable access to AFS, or don't have any access at all, then give an invalid name in ThisCell. This will cause the cache manager to initialize itself enough to store tokens, but not enough to mount /afs. BosConfig must contain no instances. It's ok to just delete it if it exists. Client binaries for bos, and pts are also required, and the kernel extension should be loaded (but afsd need not be run). KeyFile contains the key for "afs". Setting this is a bit non-obvious. use kadmin.local to create afs: # /usr/krb5/sbin/kadmin.local kadmin.local: ank -randkey afs Get the key. kadmin.local: ktadd -e des-cbc-crc:v4 -k /tmp/afs.kt afs kadmin.local: exit ktadd will report the kvno. Note it for following. # /tmp/asetkey add 3 /tmp/afs.kt afs ^ change 3 to actual kvno used, if not 3. Note that "ktadd" changes the key *each* time it is successfully run. It may be necessary to kill bosserver & any afs server processes running after changing the key of afs. It will be necessary to reauthenticate to afs after doing so as well, especially if the old key for afs was somehow not useful. Start bosserver if it was not running. Give admin "bos": bos adduser admin -localauth Make admin instance (in PT): pt_util -w admin 128/20 1 -204 -204 system:administrators 130/20 -204 -204 -204 admin 1 ^D Note extra space on last admin line -- that's *necessary* Also note expected errors -- expect something like: # pt_util -w pt_util: /usr/afs/db/prdb.DB0: Bad UBIK_MAGIC. Is 0 should be 354545 Ubik Version is: 2.0 admin 128/20 1 -204 -204 system:administrators 130/20 -204 -204 -204 Error while creating system:administrators: Entry for id already exists admin 1 pt_util: Ubik Version number changed during execution. Old Version = 2.0, new version = 33554432.0 # Install ptserver: bos create ptserver simple /usr/afs/bin/ptserver -localauth (these will start running when installed). Make k5 run from bos: (find and kill kadmind, krb5kdc, krb524d.) bos create kadmind simple '/usr/krb5/sbin/kadmind -nofork' -localauth bos create kdc simple '/usr/krb5/sbin/krb5kdc -n -4 nopreauth' -localauth bos create krb524d simple '/usr/krb5/sbin/krb524d -m -nofork' -localauth If cellservdb was updated since the cache manager was run, must tell the cache manager about it: fs newcell test: kinit admin klist aklog -c -k tokens STOP here if you are a weird person and file service is not necessary. Otherwise, on DB server, bos create vlserver simple /usr/afs/bin/vlserver -localauth bos create fakeka simple /usr/krb5/sbin/fakeka -localauth STOP here if you don't want to install any file services on your DB server. On one (or more) file servers running bosserver (but not DB server): do whatever is needed to make /vicepa (& /vicepb &etc if desired). [Linux loopback: once-only: dd /somewhere/file count=3000 bs=10k (adjust numbers if 30M is not right for you) losetup /dev/loop0 /somewhere/file mke2fs /dev/loop0 mkdir /vicepa mount /dev/loop0 /vicepa to detach: umount /vicepa losetup -d /dev/loop0 per-boot: losetup /dev/loop0 /somewhere/file fsck /dev/loop0 mount /dev/loop0 /vicepa adjust /somewhere/file to match desired location ] Do make sure sysid does NOT exist before starting it. Bad things will soon happen if duplicates what is on any other file server. bos create fs fs \ /usr/afs/bin/fileserver \ /usr/afs/bin/volserver \ /usr/afs/bin/salvager \ -localauth If you have writeable AFS storage elsewhere and don't plan to make To make a non-standalone-server vos create a some-random-volname -localauth On some machine with a running cache manager where you have root, go to somplace where you can write afs files, and make a mount point (if necessary, any of) fs newcell cell-name server-name kinit aklog fs mkm some-place some-random-volume -cell cellname To make a server from scratch: I. "do it yourself". vos create a root.cell vos create a root.afs Now boot some cache mangaer with the client side "ThisCell" set to point to this cell. Authenticate as admin, make a mount point for your cell "fs mkm /afs/ root.cell -cell ". make mount points for any other cells you care about. Replicate volumes as desired. II. take pre-built CD-rom contains "virgin.tgz", which contains some pre-built-volumes. vos restore a root.afs < root.afs vos restore a root.cell < root.cell & etc. (user, system.doc, system, dept, etc.) Now boot some cache mangaer with the client side "ThisCell" set to point to this cell. Authenticate as admin, and fix up mount points in /afs as necessary. Probably you will need to make /afs/ and delete /afs/linuxbox.private.nu at the very least. Replicate volumes as needed. For additional DB servers, copy KeyFile as needed. Don't install "fs" unless necessary. For just a file server, don't install vlserver or ptserver (or buserver) or krb5kdc/kadmind/krb524d Copy KeyFile from a db server. Do install "fs". Beware "sysid" - make sure it does not exist before first running the fileserver. Backups: 1. DB server configuration (once only): /usr/local/bin/bos create localhost buserver simple /usr/afs/bin/buserver -localauth /usr/local/sbin/backup -localauth addhost adogslife.gpcc.itd.umich.edu 0 addvolset adogslife.all addvolentry adogslife.all adogslife.gpcc.itd.umich.edu .* .*\.backup adddump /full 2. backup tape host configuration (once only): mkdir /usr/afs/backup put into /usr/afs/backup/tapeconfig 2g 1k /home/backup/output 0 2g 1m /dev/st0 1 also /usr/afs/backup/CFG_0 FILE YES 3. backup tape daemon startup: /usr/local/sbin/butc 0 -localauth 4. Each time, to do a backup: vos backupsys -localauth backup -localauth dump adogslife.all /full when dump completes, on tape server: mv /home/backup/output 5. other interesting options: backup -localauth listhosts listvol listdumps status dumpinfo to read a label: tape server: cd /home/backup ; ln output backup readlabel -localauth tape server: rm /home/backup/output 6. to run backup server on another host: #1 install software & std cellservdb info; then hack cellservdb, replace all servers with just that host's name. #2 create an instance of buserver on that host. #3 on real db hosts, create an instance of buredir: /usr/local/bin/bos create localhost buredir simple '/usr/afs/bin/udprelay2 reservoirdogs.ifs.umich.edu' -localauth //// to enable umich style replication Again, make sure krb5_prop and krb5_rep exist in /etc/services. If they don't, replication will NOT work. Look back earlier in these notes for the proper values. HOST1# /usr/krb5/sbin/kadmin.local kadmin.local: ank -randkey host/HOST1 kadmin.local: ank -randkey host/HOST2 kadmin.local: ank -randkey host/HOST3 kadmin.local: ktadd host/HOST1 kadmin.local: ktadd -k /tmp/host2 host/HOST2 kadmin.local: ktadd -k /tmp/host3 host/HOST3 scp /tmp/host2 host2:/etc/krb5.keytab scp /tmp/host3 host3:/etc/krb5.keytab HOST2# chown root /etc/krb5.keytab ; chmod 600 /etc/krb5.keytab HOST3# chown root /etc/krb5.keytab ; chmod 600 /etc/krb5.keytab on master machine (HOST1), edit /usr/krb5/var/krb5kdc/kadm5.acl and add host/HOST1@MY_NEW_REALM * host/HOST2@MY_NEW_REALM * host/HOST3@MY_NEW_REALM * on slave machines (HOST2, HOST3) create /usr/krb5/var/krb5kdc/kpropd.acl host/HOST1@MY_NEW_REALM host/HOST2@MY_NEW_REALM host/HOST3@MY_NEW_REALM (HOST1, HOST2, HOST3 = the kerberos kdc machines.) Make sure /etc/krb5.conf lists all 3 machines for kdc=, and only the master site for the admin site. Also copy over kdc.conf and the key stash file (.k5.$MY_NEW_REALM) to /usr/krb5/var/krb5kdc Also have to copy over: principal principal.kadm5 principal.kadm5.lock principal.ok Once the slave machines start, these files will be in /usr/krb5/var/krb5kdc .k5.TEST.GPCC.ITD.UMICH.EDU copied kdc.conf copied kpropd.acl made master_delta ( auto ) principal ( auto ) principal.kadm5 ( auto ) principal.kadm5.lock ( auto ) principal.ok ( auto ) slave_data ( auto ) slave_delta ( auto ) //// to run k5 services from afs: / on all hosts: bos create HOST fakeka simple '/usr/krb5/sbin/fakeka' -localauth bos create HOST kdc simple '/usr/krb5/sbin/krb5kdc -n' -localauth bos create HOST krb524d simple '/usr/krb5/sbin/krb524d -m -nofork' -localauth / on master or only server: bos create HOST1 kadmind simple '/usr/krb5/sbin/kadmind -nofork' -localauth / on master server: bos create HOST1 krep simple '/usr/krb5/sbin/krep -n' -localauth / on slave server: bos create HOST2 krepd simple '/usr/krb5/sbin/krepd -S' -localauth ====== vos create adogslife.gpcc.itd.umica root.cell -cell dogs vos create adogslife.gpcc.itd.umica root.afs -cell dogs cd /afs/umich.edu/user/m/d/mdw/test fs mkm dogs root.afs -cell dogs cd dogs vos create adogslife.gpcc.itd.umica group -cell dogs vos create adogslife.gpcc.itd.umica user -cell dogs vos addsite adogslife.gpcc.itd.umich.edu a root.afs vos addsite adogslife.gpcc.itd.umich.edu a root.afs -cell dogs vos addsite adogslife.gpcc.itd.umich.edu a root.cell -cell dogs vos addsite adogslife.gpcc.itd.umich.edu a user -cell dogs vos addsite adogslife.gpcc.itd.umich.edu a group -cell dogs fs mkm group group fs mkm user user fs mkm .group group -rw fs mkm .user user -rw vos release user -cell dogs vos release group -cell dogs vos release root.afs -cell dogs vos release root.cell -cell dogs fs checkv ================================== replication errors: on master site, look for: Jan 13 01:33:06 strawdogs krep[18448]: /usr/krb5/sbin/krep: Caught broken pipe signal if there are lots of these, look back before and see i there is this: Jan 13 01:31:51 strawdogs krep[18449]: /usr/krb5/sbin/krep: Caught broken pipe signal Jan 13 01:31:51 strawdogs krep[18449]: /usr/krb5/sbin/krep: Broken pipe while sending message block (reservoirdogs.ifs.umich.edu) Jan 13 01:31:51 strawdogs krep[18449]: reservoirdogs.ifs.umich.edu: Transmission failed, backing up our file position and reconnecting... Jan 13 01:31:51 strawdogs krep[18449]: Opening connection with reservoirdogs.ifs.umich.edu Jan 13 01:31:51 strawdogs krep[18449]: Authenticating with reservoirdogs.ifs.umich.edu Jan 13 01:31:51 strawdogs krep[18449]: /usr/krb5/sbin/krep: Ticket expired while authenticating to server (reservoirdogs.ifs.umich.edu) Jan 13 01:32:06 strawdogs krep[18448]: /usr/krb5/sbin/krep: Ticket expired while authenticating to server (backyarddogs.ifs.umich.edu) Jan 13 01:32:06 strawdogs krep[18449]: /usr/krb5/sbin/krep: Ticket expired while authenticating to server (reservoirdogs.ifs.umich.edu) Jan 13 01:32:21 strawdogs krep[18448]: /usr/krb5/sbin/krep: Connection reset by peer while authenticating to server (backyarddogs.ifs.umich.edu) Jan 13 01:32:21 strawdogs krep[18449]: /usr/krb5/sbin/krep: Connection reset by peer while authenticating to server (reservoirdogs.ifs.umich.edu) Jan 13 01:32:36 strawdogs krep[18448]: /usr/krb5/sbin/krep: Caught broken pipe signal If so that means krep has been running "too long". Stop & start it. Gah.