Securing Informix database passwords using Hashicorp Vault

Contents

Introduction

An ideal situation for securing systems is the automatic rotation of passwords. Password rotation limits the time that compromised passwords could be used by a system attacker.

Another security ideal is that passwords be provided over a secure channel - to only the person or system that requires the password.

In the past we were failing these two ideals when it came to the treatment of database passwords:

Things needed to change - and they did!

Background information

IBM Informix databases are at the core of Reece Groups' IT systems. These databases are hosted on either IBM AIX or Linux operating systems.

Informix is configured to delegate the responsibility of user authentication to the operating system.

User accounts on our AIX servers are “local accounts” - that is accounts are not sourced from a central store of accounts, such as Active Directory.

User accounts on our Linux servers are sourced from Active Directory using sssd.

Securing microservice access

The first use case for Hashicorp Vault secrets management at reecetech was to change and improve how password management was done with respect to Informix databases.

We decided to aim high and implement improvements that meant:

After some consideration, we designed a scheme where each microservice would have a pair of user accounts provisioned for it to use. This scheme allows us to rotate the password of one of the user accounts, whilst the other user account of the pair is in active use. After the password is rotated, then that particular user account of the pair would be published as the new “active” user account.

The image below illustrates how each user account of the pair has its password rotated on a two-hourly basis. We set a user account as “active” immediately after rotating its password. Offsetting the password rotations by an hour gives an apparent password rotation schedule of once-hourly, however, each password is valid for two hours. This allows us to avoid a race condition where a microservice might retrieve a password immediately before a scheduled rotation.

A pair of user accounts with password rotation schedule illustrated

Implementing this scheme uses both Active Directory secrets engine and the Key/Value secrets engine. Orchestrating the scheme is a shell script executing each hour by the various AIX servers cron daemon. Each element has a particular responsibility:

The shell script is the “brains” of the operation. When it is executed it follows these steps:

  1. Read from a reference list of microservices
  2. For each microservice, read from the Key/Value secrets engine which account of the user account pair is currently “active”
  3. Retrieve the password for the currently “non-active” user account from the Active Directory secrets engine - this triggers a password rotation by the secrets engine before it returns the freshly rotated password to the script
  4. Write the new password to /etc/shadow file
  5. Write the new password along with the username of the newly “active” user account to the Key/Value secrets engine

The script authenticates to the Vault cluster using the AppRole authentication method. We have restricted the role to the static IP addresses of AIX based servers.

The script also handles adding new users to, and deleting old users from, the /etc/passwd file as microservices are added and deleted from the reference list.

Microservice integration

Microservices at reecetech needed a code update to take advantage of the passwords being provided from Vault, instead of from environment variables.

The majority of microservices at reecetech are either Spring Boot/Java or Django/Python. This allowed us to develop a library for each framework to simplify the addition of Vault based password retrieval.

The essence of the library for both Spring Boot and Django is to check with Vault for the latest username & password when a database connection is about to be opened. The Django driver is open source and available here.

The code change was then very small for most microservices, as it involved importing the new library (which in turn imported the Vault SDK) and making use of it. The best part of the change was deleting the old username & password combination from the Helm chart values, originally sourced from the passwords wiki page in the bad old days!

The microservices are deployed to Kubernetes. The Vault cluster is configured to “trust” Kubernetes as a source of identity using the Vault Kubernetes authentication method. This authentication method requires some configuration on both the Kubernetes cluster and the Vault cluster to establish the trust.

Microservice containers in our Kubernetes cluster automatically have a signed JWT token injected into them. To authenticate to Vault the container transmits the token to Vault, which then calls the Kubernetes TokenReview API to verify whether the token is legitimate. Once the container has successfully authenticated with Vault, the container is restricted by Vault policy to only being able to access its own passwords in the Key/Value secrets engine. This means that one microservice cannot access the passwords of another microservice - satisfying the ideal that passwords are provided only to the person or system that requires the password.

Other helpers

Two more components complete the solution:

Diagram

Putting all the pieces together, the solution for microservice access to Informix databases using secure and rotated passwords looks like this:

A diagram showing all the components of the solution and their responsibilities and actions

Securing people’s access

Apart from microservices, we also need people to have access to databases. Typically our people use graphical SQL software such as DBeaver or SQuirreL.

The scheme for people to have secure rotated passwords is different to the microservice scheme described above. The key differences are:

Implementing this scheme uses the “Service Account Check Out” (or “library”) feature of the Active Directory secrets engine in Vault. The library feature provides several user accounts to be “borrowed” by people as they require them.

The password for any user account is not revealed until the account is borrowed from the library - and then it is only shown to the borrower. A user account has its password rotated on “check-in” (returned to the library) so that the person who had just returned the user account can no longer use the account.

The complimentary piece of this solution is an auditing service. The auditing service is implemented as an AWS Lambda function which inspects the status of the library at regular intervals. The function then maps borrowed user accounts back to the real people who had borrowed the account. It stores this information in an AWS DynamoDB table for later querying.

We require this auditing service as the user accounts borrowed from the library are not tied to any particular person - but we want to be able to find out who did what in the database in case of things going wrong (e.g. an intensive query that uses all the CPU).

The auditing service will also post a message to Slack when a production database credential is borrowed. Accessing production databases is discouraged, but is allowable for emergency purposes. The Slack message looks like this:

Slack message notifying of production password borrowing

Customisation for AIX

We needed a way for our AIX servers to synchronise passwords to /etc/shadow, and thus to be able to see what the passwords for the library user accounts were before they are borrowed. Vault is open source, and thus we could modify the Active Directory secrets engine for our own particular need. The modified code is here, and we do not intend to merge it back upstream as it is a temporary solution for reecetech. Eventually, all our database servers will be Linux based and integrated with Active Directory.

The customised Active Directory secrets engine exposes an additional endpoint /creds which allows AIX based machines to see what the current password is for all the user accounts in the library. This endpoint is heavily restricted by Vault policy which allows only the AIX AppRole access.

A shell script is executed on each AIX server to synchronise the passwords. When it is executed it follows these steps:

  1. Read from a reference list of library user accounts
  2. For each user account, read from the custom /creds endpoint to obtain the current password
  3. Write the new password to /etc/shadow file

The script authenticates to the Vault cluster using the AppRole authentication method. We have restricted the role to the static IP addresses of AIX based servers.

The script also handles adding new users to, and deleting old users from, the /etc/passwd file if the library is expanded or reduced.

User experience

At reecetech we have written a CLI utility called vampire-squid that assists with Vault related actions. It’s helpful as it orchestrates things such as obtaining tokens from our identity provider and then generating a session token in Vault for the users.

For a person to obtain a username & password for an Informix database from Vault, they use vampire-squid like so:

# vampire-squid informix -d example
INFO: No token file found at ~/.vault/nonprod
INFO: Authenticating to Vault nonprod cluster as "user"

Please login using your standard user account
Complete the login via your OIDC provider. Launching browser to:

    https://oidc.provider.example.com/oauth2/v1/authorize?redirect_uri=http%3A%2F%2Flocalhost%3A8250%2Foidc%2Fcallback

The person then follows the link in their browser to complete the authentication sequence. If authenticated they are notified in the browser and head back to the terminal.

Successful authentication sequence reported in a browser window

Back in the terminal vampire-squid has received the authentication details and continued on to borrow a user account from the library:

...


INFO: Token policies: ['default', 'informix/example', 'users/base', 'users/team_example_team']

Your credentials for the example Informix database is:
  Username: ExampleUA1002
  Password: ?@examplecreds8eD5Fs05zytheydonotactuallyworknNzcTMexamplecreds

These credentials are valid for 12 hours

At this point, the person can copy and paste this username & password into the SQL client software. The user & password is also written to the persons' home directory. We expect some future tooling will take the written copy and automatically use it in the SQL client software - using squirrelcli for SQuirreL for example.

Other helpers

Two more components complete the solution:

Diagram

Putting all the pieces together, the solution for people’s access to Informix databases using secure and rotated passwords looks like this:

A diagram showing all the components of the solution and their responsibilities and actions

Wrapping up

Despite the challenges of operating in what some might regard as a legacy system - Informix on AIX - and that system that had an established pattern of practises that don’t meet modern security expectations, we’ve been able to significantly upgrade the security posture of the system. Not only that we’ve also provided a solution that is cross-compatible with Informix on Linux based servers.

Operating Hashicorp Vault is not easy, but with the right focus, it can be incredibly valuable in improving security on systems that might seem to be a lost cause. Vault’s easy to use RESTful HTTP interfaces make it straightforward to integrate with anything - including ksh93 shell scripts on AIX based servers using nothing else but cURL & jq.