March 17, 2015

IBM Spectrum Scale Object Encryption at Rest

Hi Everyone,

I guess you would have reached my blog while searching for help related to setting up file level encryption in Spectrum Scale (formerly GPFS) and I hope you will not be disappointed.

The configuration steps shown here are much similar to the scenario explained in knowledge center. But I just thought to represent the configuration steps in pictorial representation for easy understanding and faster configuration.

As a prerequisite for this setup we would require;
- A RKM (Remote Key Management) sever, currently IBM Security Key Lifecycle Manager (ISKLM) version or later is only supported
- A GPFS cluster which runs Advanced Edition (v4.1)

Once you have both of these ready, you are good enough to configure file level encryption.

(Note: click on the posted screenshots for a full screen view of operations)

Step-1: Just as a validation step, ensure the ISKLM server version is >= (For install and upgrade instructions of ISKLM refer to its product guides)

The ISKLM login via web browser can be made using https://[ISKLM-address]:9080/ibm/SKLM/login.jsp

On the right side top of ISKLM GUI, you can view “About” tab which displays a window showing current version.

Step-2: Create a self signed SSL / KMIP Server certificate

Step-3: Validate the below entries populated in, else manually fill in (fips=on is only required if you want to adhere FIPS 140-2 compliance)

[root@isklm ~]# cat /opt/IBM/WebSphere/AppServer/products/sklm/config/
#Mon Mar 16 20:21:58 IST 2015

Step-4: Restart ISKLM server. Once the restart completes , you should be able to see SSL protocol, KMIP protocol status as “configured”

Step-5: Validate whether the created self-signed certificate is configured properly and is in-use

Step-6: Export the configured / in-use server certificate as a file, follow the CLI commands shown below (store this certificate file, it needs to be copied to GPFS nodes)

[root@isklm ~]# cd /opt/IBM/WebSphere/AppServer/bin/
[root@isklm bin]# ./ -username SKLMAdmin -password Passw0rd -lang jython
WASX7209I: Connected to process "server1" on node SKLMNode using SOAP connector;  The type of process is: UnManagedProcess
WASX7031I: For help, enter: "print"
wsadmin>print AdminTask.tklmCertList('[-alias cert1_label]')
CTGKM0001I Command succeeded.

uuid = CERTIFICATE-c00f107e-6970-44ff-9225-09c86c17dd85
alias = cert1_label
key store name = defaultKeyStore
key state = ACTIVE
issuer name = CN=cert1
subject name = CN=cert1
creation date = 3/16/15 2:02:13 PM India Standard Time
expiration date = 3/15/18 2:02:13 PM India Standard Time
serial number = 657606986627

wsadmin>print AdminTask.tklmCertExport('[-uuid CERTIFICATE-c00f107e-6970-44ff-9225-09c86c17dd85 -format base64 -fileName /root/srvcert]')
CTGKM0001I Command succeeded.

Step-7: Create a new device group and choose the Device family as “GPFS”

Step-8: Create Keys associated to the device group created in the above step

You can select the option “Hold new certificate requests pending my approval”, for manual validation of the client.

Copy the keys (for example "KEY-c2cf0496-ded9-42d3-b341-d0ace97fcdcf" is used in step10), and it will be used by GPFS encryption policy 

Step-9: Create a keystore using the ISKLM certificate exported in step-6

[root@eso1 ~]# mkdir /var/mmfs/etc/RKMcerts
[root@eso1 ~]#
[root@eso1 ~]# ls -lrth /root/srvcert
-rw-r--r--. 1 root root 1.1K Mar 16 18:56 /root/srvcert
[root@eso1 ~]#
[root@eso1 ~]# mmauth gencert --cname GPFS_TENANT1 --label client_label --cert /root/srvcert --out /var/mmfs/etc/RKMcerts/ISKLM.p12 --pwd client_label
[root@eso1 ~]#
[root@eso1 ~]# ls -lrth /var/mmfs/etc/RKMcerts/ISKLM.p12
-rw-------. 1 root root 4.0K Mar 17 11:55 /var/mmfs/etc/RKMcerts/ISKLM.p12
[root@eso1 ~]#
[root@eso1 ~]# cat /var/mmfs/etc/RKM.conf
ISKLM_srv {
  type = ISKLM
  kmipServerUri = tls://
  keyStore = /var/mmfs/etc/RKMcerts/ISKLM.p12
  passphrase = client_label
  clientCertLabel = client_label
  tenantName = GPFS_TENANT1

Step-10: Setup an encryption policy and apply it to the file system (device) to which you wanted to enable encryption

[root@eso1 ~]# cat enc_policy
RULE 'p1' SET POOL 'system'
RULE 'Encrypt all files in filesystem with rule E1'
RULE 'simpleEncRule' ENCRYPTION 'E1' IS
[root@eso1 ~]#
[root@eso1 ~]# mmchpolicy gpfs-enc enc_policy -I yes
Validated policy `enc_policy': Parsed 3 policy rules.
Policy `enc_policy' installed and broadcast to all nodes.

Step-11: File I/O failure as the client device is not yet trusted in ISKLM Server

(Reported error - Key could not be fetched)

Step-12: Accept the client device communication certificate

Step-13: File I/O failure as a result of wrong key

Here in this example, I have used a dummy key "KEY-ffc98c44-e8d8-4744-88da-1ca6322c9c4a" which is not associated with the GPFS device group "GPFS_TENANT1".

(Reported Error - Permission denied)

Step-14: File I/O success using correct key and its encryption attributes

(Here gpfs-enc is the filesystem to which we have loaded encryption policy  - step10 and gpfs1 is an unencrypted filesystem)

Step-15: Install and Configure Spectrum Scale Object with the help of  red paper and use gpfs_mount_point as "/gpfs-enc" for object storage path

That all, you now have an encryption configured High Performance Scalable Object Storage....


  1. Nice work Sasi! Will you also be publishing to Developer Works?

  2. This comment has been removed by the author.

  3. Thanks for sharing this info on encryption configured High Performance Scalable Object Storage.... if anyone is looking for End to End Security with Encrypted CHAT Service download this App from Google Play URL:

  4. Hi Sasi,

    We are trying GPFS to SKLM integration but facing exception while executing step #11. I am getting the following exception:

    The key server (port 6000) had a failure and will be quarantined for 1 minute(s).

    Unable to create encrypted file test (inode 12295, fileset 0, file system gpfs0).

    Key 'KEY-3f273419-180e-443d-8c4a-61ccb308cd6e:ISKLM_srv' could not be fetched. The TLS handshake could not be completed successfully.

    Can you please guide on the steps which we may be missing ?

    Thanks in Advance,

    Mohit Agrawal,

  5. Hi Mohit,

    If you are still stuck with the above error run following command to check connectivity to the SKLM server -
    openssl s_client -connect

    And if connectivity is okay, then check the following variable

    cat /mnt/ISKLM/opt/IBM/WebSphere/AppServer/products/sklm/config/| grep TransportListener.ssl.protocols

    Changing the above parameter will need a server restart

  6. configured gpfs without using websphere ... always better to avoid guis ...
    but after it worked for a few months, appears a cert expired or otherwise became invalid ...
    cant figure out how to repair this ...

    Tue Feb 14 13:56:44 EST 2017: mmaddnode: Processing node
    /usr/lpp/mmfs/bin/tsgskkm error: could not insert the key in the keystore (error 48).
    mmremote: Unexpected error from tsgskkm store --cert /var/mmfs/ssl/stage/tmpKeyData.mmremote.28876.cert --priv /var/mmfs/ssl/stage/tmpKeyData.mmremote.28876.priv --out /var/mmfs/ssl/stage/tmpKeyData.mmremote.28876.keystore. Return code: 255
    mmremote: The CCR environment could not be initialized on node
    mmaddnode: The CCR environment could not be initialized on node
    mmaddnode: mmaddnode quitting. None of the specified nodes are valid.
    mmaddnode: Command failed. Examine previous error messages to determine cause.

    1. This comment has been removed by the author.

  7. What are the expiration dates on your certs/keys? add your local username/password of course.

    /opt/IBM/WebSphere/AppServer/bin/ -username username -password password -lang jython -c 'print AdminTask.tklmCertList ()'

    /opt/IBM/WebSphere/AppServer/bin/ -username username -password password -lang jython -c 'print AdminTask.tklmKeyList ()'

    Did you remember to copy the /var/mmfs/etc/RKM.conf and /var/mmfs/etc/RKMcerts directory to the new machines? These are NOT added to CCR and require manually adding them on new machines. (at least up to