Logs and logging

There are 3 types of logs in SaunaFS

Metadata logs

Each change in the filesystem is being logged Default location for those loges is at /var/lib/saunafs/

#root@tst1-builder-01 /var/lib/saunafs # ls -lah /var/lib/saunafs/*log*
-rw-r----- 1 saunafs saunafs 241 Jan  4 13:42 /var/lib/saunafs/changelog.sfs
-rw-r----- 1 saunafs saunafs 13M Jan  3 02:04 /var/lib/saunafs/changelog.sfs.35
-rw-r----- 1 saunafs saunafs 25M Jan  3 01:59 /var/lib/saunafs/changelog.sfs.36
-rw-r----- 1 saunafs saunafs 427 Jan  2 19:30 /var/lib/saunafs/changelog.sfs.42
-rw-r----- 1 saunafs saunafs 18K Jan  2 18:59 /var/lib/saunafs/changelog.sfs.43
-rw-r----- 1 saunafs saunafs 37M Dec 23 06:19 /var/lib/saunafs/changelog.sfs.50

Below we can se an example of empty file creation:

And followed by making snapshot of this emplty file:

tail /var/lib/saunafs/changelog.sfs

5504749: 1704375765|CREATE(1,example_empty_file,f,436,1008,1997,0):3145732
5504750: 1704375765|ACQUIRE(3145732,4)
5504751: 1704375765|ATTR(3145732,436,1008,1997,1704375765,1704375765)
5504752: 1704375765|CHECKSUM(4.0.0):12158089599274070817
5504753: 1704375786|RELEASE(3145732,4)
✓ (0.00080s) 13:46:11

tail /var/lib/saunafs/changelog.sfs

5504749: 1704375765|CREATE(1,example_empty_file,f,436,1008,1997,0):3145732
5504750: 1704375765|ACQUIRE(3145732,4)
5504751: 1704375765|ATTR(3145732,436,1008,1997,1704375765,1704375765)
5504752: 1704375765|CHECKSUM(4.0.0):12158089599274070817
5504753: 1704375786|RELEASE(3145732,4)
5504754: 1704376084|CLONE(3145732,1,3145733,example_empty_file.snapshot,0)

This logging is giving a powerful ability to apply Realtime or offline analysis (e.g security, usage, anomalies, etc)

Thosche changelog files also include checksum (which could be use to determine manipulation of logs data) used to detect potential errors in logs,

Copy of those changelogs can be stored in dedicated meta logger server in cluster for having a copy.

Regular syslog (journalctl)

In the confiuration file for every server/daemon/service

/etc/saunafs/sfsmaster.cfg

We have posibility to define syslog ID

Default for master server is:

# SYSLOG_IDENT = sfsmaster

sudo journalctl --since "3 minutes ago" | grep sfsmaster

Client site operation logs (oplog)

In case we need to determine/monitor whtat is happening in particular mountpoint we can access

sudo cat /<MOUNT_POINT>/.oplog

For example

sudo grc cat /mnt/sfs.208.29421/.oplog
1704376778 01.04 13:59:38.838828: uid:0 gid:2147483651 pid:395006 cmd:open (4294967281) (internal node: OPLOG): OK (1,0)
1704376778 01.04 13:59:38.838934: uid:0 gid:2147483651 pid:395006 cmd:getattr (4294967281) (internal node: OPLOG): OK (3600,[-r--------:0100400,1,0,0,0,0,0,0])

Example touch

touch example_empty_file

1704376852 01.04 14:00:52.800360: uid:0 gid:2147483651 pid:396488 cmd:getattr (1): OK (1.0,[drwxrwxrwx:0040777,4,0,0,1704376839,1704376084,1704376084,0])
1704376852 01.04 14:00:52.804180: uid:0 gid:2147483651 pid:396490 cmd:lookup (1,example_empty_file): OK (0.0,3145732,1.0,[-rw-rw-r--:0100664,1,1008,1997,1704375765,1704375765,1704375765,0])
1704376852 01.04 14:00:52.804646: uid:0 gid:2147483651 pid:396490 cmd:getxattr (3145732,system.posix_acl_access,4096): Attribute not found
1704376852 01.04 14:00:52.805048: uid:0 gid:2147483651 pid:396490 cmd:open (3145732): OK (0,0)
1704376852 01.04 14:00:52.805231: uid:0 gid:0 pid:396490 cmd:flush (3145732): OK
1704376852 01.04 14:00:52.805663: uid:0 gid:2147483651 pid:396490 cmd:setattr (3145732,0x1B0,[---------:00000,0,0,1704376852,1704376852,0]): OK (1.0,[-rw-rw-r--:0100664,1,1008,1997,1704376852,1704376852,1704376852,0])
1704376852 01.04 14:00:52.805764: uid:0 gid:0 pid:396490 cmd:flush (3145732): OK
1704376852 01.04 14:00:52.805817: cmd:release (3145732): OK