DataHub
Qry reads DataHub's descriptions, glossary terms, tags, lineage and usage to answer better, and writes column descriptions there when someone approves them in Readiness. What it does with DataHub is described in DataHub Integration; this page is how to connect it and keep it connected.
The account Qry uses
Qry calls DataHub with one access token, and DataHub records everything Qry does under that token's account. Use a service account, not a person's:
- In DataHub, Settings › Service Accounts › Create. Name it
qry. - Give it the Editor role (Settings › Users & Groups › Roles). Qry needs to read metadata and to edit descriptions; Editor covers both. Reader is enough if nobody will approve descriptions into DataHub.
- Give it the Generate Personal Access Tokens privilege, with a platform policy (Settings › Permissions › Policies › Create, type Platform, privilege Generate Personal Access Tokens, applied to the
qryaccount). This is what lets Qry renew its own token — see below. - Create a token for the account (Settings › Service Accounts › qry › Create token) with the longest expiry your DataHub allows.
Then in Qry, Admin › System configuration › DataHub: turn the integration on, enter the GMS URL (for example http://datahub-gms:8080) and paste the token. Test checks that DataHub accepts it and says which account it acts as.
A token of type service account can only be created by an account with Manage Service Accounts, which also creates and deletes accounts. A personal token for the qry account itself needs only Generate Personal Access Tokens, and acts as that same account. That is the narrower grant, and the one Qry uses.
Tokens expire, and Qry renews its own
From DataHub 1.6 on, every new token has an expiry. When Qry's expires nothing fails loudly: answers simply stop using DataHub's descriptions, and approving a description into DataHub fails. So Qry renews its token itself.
Every day (05:00 UTC), when fewer than 30 days are left, Qry:
- asks DataHub, with the token it has, for a new personal token for the same account — for a year, or the longest your DataHub allows;
- checks that the new token works and acts as the same account — if it does not, it revokes it and keeps the current one;
- saves it in place of the old one and reloads every Qry worker;
- the next day, revokes the token it replaced, if Qry created it. The first token, created by an administrator, is not Qry's to revoke: it simply expires.
A DataHub page left open from before a renewal still holds the replaced token. Saving it keeps the renewed token — any other change on the page is saved — so it cannot put back a token Qry has revoked. A token you paste yourself is always saved as is.
If the renewal fails — most often because the account lacks Generate Personal Access Tokens — the current token stays, the reason is shown on the DataHub page, and administrators get a notification every day until it is fixed.
The DataHub page shows when the token expires, how many days are left, when Qry last renewed it and the last error, with Renew now to renew at once (whatever the expiry). Renew the token automatically turns the daily renewal off; Renew now still works.
The token is encrypted
The DataHub token is stored encrypted in Qry's database, with the same key as datasource credentials (derived from the installation's JWT_SECRET_KEY). A database backup, or someone reading the configuration table, does not get a usable token. Upgrading encrypts a token saved by an earlier version.
This matters more here than for other settings: a token whose account may create tokens can be used to create new ones, so a copy would not die with the original.
If JWT_SECRET_KEY changes — or the configuration was copied from an installation with another key — the stored token can no longer be decrypted. DataHub stops being used, the DataHub page says why, and qry doctor reports it. Paste the token again on the DataHub page.
Tokens Qry did not create
Every day Qry also lists the tokens its account owns (DataHub lets an account list its own) and looks for any it did not create. The first time it looks at an account, the tokens it already has are not reported as alarms, but administrators get one notice listing them to confirm they are theirs; a token already reported is never accepted this way, and pasting a token for another account starts over. Tokens that appear afterwards are reported. Someone holding a copy of Qry's token could create one; nobody else can, since the service account cannot sign in. If it finds one, the DataHub page lists it in red and administrators are notified daily.
Qry does not revoke these itself. If nobody on your team created them:
- revoke them in DataHub (Settings › Access Tokens);
- press Renew now in Qry, so the token that may have been copied is replaced (and revoked the next day if Qry created it; otherwise revoke it in DataHub too).
Tokens created for the account by an administrator belong to that administrator, not to the account, so they do not show here.
Each Qry installation knows only the tokens it created. If two installations — production and a test environment, say — use the same DataHub account, each reports the other's tokens as not created by Qry. Give each installation its own service account (qry, qry-test…).
Watching it from outside: qry doctor
The operator CLI checks it with datahub-token-expiring: it fails when DataHub is configured and the token has 21 days or fewer left (Qry starts renewing at 30, so this means renewal has not worked for over a week), the last renewal failed, or the account has tokens Qry did not create.
qry doctor --plane product
Troubleshooting
| What you see | What to do |
|---|---|
| DataHub refused to create a token … grant it the 'Generate Personal Access Tokens' privilege | Add the platform policy above for the qry account, then Renew now. |
| The current DataHub token is not accepted | The token expired or was revoked (or DataHub's signing key changed). Create a new one for the account and paste it on the DataHub page. |
| This token does not expire | A token from before DataHub 1.6. It keeps working until DataHub's signing key changes; renewing replaces it with one that expires and is renewed. |
| The new token acts as another account | The token you pasted is not the qry account's. Create one for the service account. |
| Days left keeps going down after a renewal | Check the Celery beat and worker pods are running: the renewal is a scheduled task. |