Proxy handling

Proxy handling in Pilot 3

This page describes which X.509 proxies the pilot handles, where they come from, what uses them, how they are verified, renewed and cleaned up, and how they are given to ALRB containers. It reflects pilot 3.14.5.4 (pilot3-rc) and the ATLAS user plugin (pilot/user/atlas). Other experiment plugins implement the same interface hooks but do not download job proxies.

Mid-job renewal of the job proxy was added so that user jobs can run longer than the lifetime of the proxy downloaded at job start (96 hours).

Contents

• The two proxies
• Where the downloads come from
• Pilot proxy
• Job proxy
• Renewal of the job proxy
• Proxies inside ALRB containers
• Derived copies
• Cleanup and security
• Configuration
• Error codes
• Testing the renewal
• Known limitations and open items

The two proxies

	Pilot proxy	Job proxy
Purpose	The pilot's own credential	Keeps the production role away from user payloads
Initial origin	Provided at pilot start by Harvester / the batch system (outside the pilot)	Downloaded by the pilot at job validation
Download URL	Renewal only: {PANDA_SERVER_URL}/api/v1/creds/get_proxy?role=atlas:/atlas/Role=production	{PANDA_SERVER_URL}/api/v1/creds/get_proxy?role=atlas&dn=atlpilo2
Legacy fallback URL	{PANDA_SERVER_URL}/server/panda/getProxy	{PANDA_SERVER_URL}/server/panda/getProxy
Identity	Production role	Pilot robot identity (atlpilo2), no production role (/atlas/Role=NULL)
When present	Always	User jobs on analysis queues with a pilot container, and user jobs on unified dispatch queues (see Job proxy)
File location	Outside the job work directory, e.g. /pool/condor/dir_2724986/x509up_u25606_prod	Job work directory
Checked	Before each job download (at least 72 h required); every 10 min during a job	Every 10 min while the payload is running (on unified dispatch queues also once before stage-out)
Renewed mid-job	Yes (job monitor, see Pilot proxy)	Yes (see Renewal of the job proxy)

Where the downloads come from

All proxy downloads go to the PanDA server credentials API. The request is a GET with the parameters shown in the table above.

PANDA_SERVER_URL is set at pilot start from the --url and --port options, defaulting to the [Pilot] pandaserver value in pilot/util/default.cfg:

pandaserver: https://pandaserver.cern.ch:25443

The alias pandaserver.cern.ch is replaced by a randomly chosen node resolved through DNS, for basic load balancing. A job proxy download on CERN-PTEST therefore looks like this in the pilot log:

url = https://aipanda090.cern.ch:25443/api/v1/creds/get_proxy, secure = True, compressed = True, ipv = IPv6, method = GET
params = {'role': 'atlas', 'dn': 'atlpilo2'}
loaded X.509 client cert/key from '/pool/condor/dir_2724986/x509up_u25606_prod'

Authentication. The request is always authenticated with the pilot proxy as X.509 client certificate. Proxy downloads never use OIDC tokens, even on queues where tokens are enabled for other server communication. A renewal of the pilot proxy therefore has to succeed while the current pilot proxy is still valid.

The VOMS role is chosen from the proxy type:

Requested role	role parameter	Extra parameters
Production (pilot proxy renewal)	atlas:/atlas/Role=production	none
User (job proxy)	atlas	dn=atlpilo2

Failure handling. If the new endpoint does not answer, the pilot falls back to the legacy endpoint /server/panda/getProxy. Transient network errors are retried, 10 s apart, for up to 3 attempts in total. Server-side failures are not retried.

Server-side cache. The server serves proxies from a cache: a job proxy downloaded at 12:58 already had timeleft 94:56:23, i.e. it had been issued about an hour earlier. The PanDA server team has confirmed that a served proxy always has at least 72 hours left. Repeated downloads can therefore return the same proxy, which the job proxy renewal takes into account (see Renewal of the job proxy).

Pilot proxy

Used for:

• PanDA server communication, including all proxy downloads. The proxy file is re-read on every request, so a renewed proxy is picked up immediately.
• Stage-in on all queues, for all jobs.
• Stage-out on all queues, except the output stage-out of user jobs on unified dispatch queues (see Job proxy).
• Log stage-out everywhere.
• The remote file-open check, except for user jobs on unified dispatch queues.

When stage-in/out or the file-open check runs in an ALRB container, the container gets a temporary copy of the pilot proxy in the work directory rather than the pilot proxy itself (see Proxies inside ALRB containers).

• Rucio traces.
• The payload of production jobs, and of user jobs that do not get a job proxy (analysis queue without a pilot container, or payload proxy verification disabled).

Verification:

• Before each job download, the pilot proxy must have at least 72 h left. Otherwise the pilot does not ask for a new job (error PROXYTOOSHORT).
• While a job is being monitored, the proxy is checked every proxy_verification_time seconds (default 600). arcproxy runs once and later checks are served from a cache.

Renewal (existing mechanism):

• When less than 40 minutes remain (1 h minus 20 min grace), the job monitor downloads a new production proxy and writes it over the proxy in use.
• If the download fails, the payload is killed (NOVOMSPROXY).
• Renewal only happens while a job is being monitored. Between jobs, the 72 h requirement applies instead.

The limitations of this mechanism are listed under Known limitations and open items.

Job proxy

A user payload must not run with the production role. The pilot therefore downloads, once per user job, a proxy of the pilot robot identity without the production role, and gives it to the payload instead of the pilot proxy.

The credential is the same on all queue types: the same request, the same identity and the same lifetime. A job gets at most one job proxy. The pilot implements it in two variants, which differ only in the queue type they apply to and in what else uses the proxy besides the payload:

	Analysis queue variant	Unified dispatch variant
Applies to	User jobs on non-unified analysis queues	User jobs on unified dispatch queues
Downloaded by	handle_payload_proxy()	handle_proxy() → download_new_proxy()
Recorded in	payload proxy path in the pilot cache	X509_UNIFIED_DISPATCH
File name	<basename>-payload.proxy	<basename>-unified.proxy
Name in logs and cache	payload	unified
Payload container	Yes	Yes
Remote file-open check	No (pilot proxy)	Yes
Stage-in	No (pilot proxy)	No (pilot proxy)
Output stage-out	No (pilot proxy)	Yes

<basename> is the file name of the pilot proxy without any .proxy suffix, e.g. x509up_u25606_prod-payload.proxy.

The different stage-out behaviour of the two variants is intentional.

Analysis queue variant

Downloaded: once per job at job validation, and only when all of the following hold:

• The job is a user (analysis) job.
• The queue type is not unified.
• The queue uses a pilot container.
• Proxy verification and payload proxy verification are both enabled.
• payload_proxy_from_server is True in default.cfg.

If the download or verification fails, the job fails at validation with PAYLOADPROXYDOWNLOADFAILURE (1391).

Used for: everything inside the payload container, including direct I/O.

Unified dispatch variant

On unified dispatch queues, the pilot serves both production and user jobs and starts with a production proxy. For each user job, the job proxy is downloaded at job validation and recorded in X509_UNIFIED_DISPATCH.

Used for:

• The payload container.
• The remote file-open check container.
• Output stage-out. For the duration of the stage-out, the pilot switches X509_USER_PROXY to the job proxy and switches it back afterwards.

Because the stage-out may take long, and the job monitor stops checking the job proxy once the job has left the running state, the job proxy is checked (and renewed if needed) once more right before stage-out. It is removed at the end of the output stage-out.

Common to both variants

The job proxy is:

• renewed as described in Renewal of the job proxy;
• given to the payload container as described in Proxies inside ALRB containers, so that the running payload sees each renewal;
• removed from the work directory before the log tarball is created (see Cleanup and security).

Renewal of the job proxy

The job proxy is renewed by the ATLAS user plugin itself. The generic renewal of the pilot proxy in the job monitor would download a production proxy, which must never replace the job proxy.

Periodic check:

1. Every proxy_verification_time seconds (default 600) while the job is in state running, the job monitor asks the user plugin to verify the job proxy.
2. The validity is read from the arcproxy cache. arcproxy runs exactly once per downloaded proxy, at download time. Both the certificate and the VOMS attribute end times are considered, and the earlier one counts.
3. If less than 24 hours remain, the proxy is renewed, unless a recent attempt has postponed the next one (see below).

A job proxy has at least 72 hours left at download, so jobs shorter than about 48 hours never renew it. A job running for 8 days needs at most three renewals.

Renewal steps:

1. A new proxy with the user role is downloaded to <proxy>.tmp next to the proxy in use.
2. The new proxy is verified with arcproxy, which also refreshes the cache entry.
3. The new proxy is only used if it is valid longer than the current one. Since the server serves proxies from a cache, it may return the very proxy already in use.
4. It is then moved over the proxy in use. The move is a single atomic replacement in the same directory, so a reader never sees a partial file.

Checks on a downloaded proxy. Every job proxy, at job start and at each renewal, is compared with the minimum validity guaranteed by the server (72 hours, with 20 minutes grace). A shorter proxy is reported with a warning (the PanDA server returned a payload proxy with only ...h left (expected at least 72h) - using it anyway), but it is still used. It may well be long enough for the job, and the problem is on the server side.

Outcome of a renewal attempt:

Outcome	What happens	Next attempt
New proxy valid for at least 24 h	Installed	Only when it drops below 24 h
New proxy valid longer than the current one, but still below 24 h	Installed	After 1 hour
New proxy not valid longer than the current one (e.g. the same proxy)	Discarded, current proxy kept	After 1 hour
New proxy with unknown validity	Discarded, current proxy kept (installing it would disable further renewals)	After 1 hour
Download, verification or replacement failed	Current proxy and its cached validity left untouched	At the next check

The 1 hour backoff does not apply in the last hour before the hard floor. There, every check makes another attempt.

Hard floor. The job is only failed once the proxy has less than 10 minutes left and could not be renewed. The pilot then records PAYLOADPROXYDOWNLOADFAILURE (1391) and kills the payload.

Safeguards:

• Only one renewal runs at a time. The job monitor and stage-out threads share a lock. A proxy renewed by the other thread while waiting is not renewed again, nor is it downloaded again if the other thread has just attempted a renewal.
• If the job has moved on while a proxy was being renewed (e.g. stage-out already removed it), the renewed file is removed again.
• A stale cache entry and any backoff from a previous job are cleared before a newly downloaded job proxy is verified.

flowchart TD
    A[Job monitor, every proxy_verification_time s] --> B{job.state == running?}
    B -- no --> Z[skip]
    B -- yes --> C{cached validity known?}
    C -- no --> Y[warn once, cannot verify or renew]
    C -- yes --> D{less than 24 h left?}
    D -- no --> Z
    D -- yes --> K{postponed by an earlier attempt?<br/>not in the last hour before the floor}
    K -- yes --> Z
    K -- no --> E[download new user proxy to proxy.tmp]
    E --> F{download and verification OK?}
    F -- no --> H{less than 10 min left?}
    F -- yes --> L{valid longer than current proxy?}
    L -- yes --> G[atomically replace proxy in use]
    G --> M{still below 24 h?}
    M -- yes --> P[postpone next attempt by 1 h]
    M -- no --> Z
    L -- no --> Q[keep current proxy, postpone next attempt by 1 h]
    Q --> H
    H -- no --> I[keep current proxy, retry later]
    H -- yes --> J[error 1391, kill payload]

Loading

Proxies inside ALRB containers

What ALRB does with X509_USER_PROXY. When ALRB starts a container, it copies the proxy given by X509_USER_PROXY into its own directory and points X509_USER_PROXY inside the container to that copy. This happens whether the variable is exported in the container command or merely inherited from the pilot's environment. ALRB's directory is under ALRB_CONT_CHOME, which the pilot sets to the job work directory. This has two consequences:

• A renewed proxy is not seen inside the container, since the container reads ALRB's copy.
• A copy of whatever proxy is in X509_USER_PROXY, including the more privileged pilot proxy, ends up under the job work directory. That directory is visible to the user payload as /srv and is archived in the log tarball, while the ATLAS redundant-file cleanup only looks at its top level.

What the pilot does instead, following the ALRB maintainers' recommendation:

1. The proxy the container needs is placed directly in the job work directory, which is mounted as /srv inside the container.

2. The pilot writes a small presetup script into the work directory, which points X509_USER_PROXY to that proxy as seen inside the container. The script contains only a path, never the proxy itself.

3. The container command unsets X509_USER_PROXY, so that ALRB has nothing to copy, and sets ALRB_CONT_PRESETUP to source the script:

   unset X509_USER_PROXY;export ALRB_CONT_PRESETUP="source /srv/pilot_proxy_presetup.sh";...;source ${ATLAS_LOCAL_ROOT_BASE}/user/atlasLocalSetup.sh -c ...
   

4. Inside the container, ALRB evaluates ALRB_CONT_PRESETUP as a command before its own setup, and before the payload or stage-in/out script runs.

Each container type has its own script, so that a script written for one container never affects another:

Container	Proxy given to it	Presetup script
Payload, with a job proxy	The job proxy in the work directory, renewed in place	pilot_proxy_presetup.sh
Payload, without a job proxy (production jobs, user jobs without job proxy)	Pilot proxy, exported as before (ALRB copies it)	none
Stage-in	Temporary copy of the pilot proxy	pilot_proxy_presetup_stage-in.sh
Stage-out	Temporary copy of the pilot proxy; on unified dispatch queues, the job proxy for the output stage-out	pilot_proxy_presetup_stage-out.sh
Remote file-open check	Temporary copy of the pilot proxy; on unified dispatch queues, the job proxy	pilot_proxy_presetup_file-open.sh
Containers started without a proxy (debug command)	None; X509_USER_PROXY is unset so the inherited pilot proxy is not copied	none

Example script for the payload container:

#!/bin/bash
# Created by the PanDA pilot. Sourced inside the container via ALRB_CONT_PRESETUP, so that the
# container uses the proxy in /srv (for the payload, renewed by the pilot during the job).
export X509_USER_PROXY=/srv/x509up_u25606_prod-payload.proxy

Temporary copies for stage-in/out and the file-open check. These containers run pilot code, not user code, and need the pilot proxy, which lives outside the work directory. Just before such a container is started, the pilot copies the pilot proxy to <basename>-container-<label>.proxy in the work directory, e.g. x509up_u25606_prod-container-stage-in.proxy. The copy is only readable by its owner. It is removed as soon as the container command returns, whether it succeeded, failed or was killed on a timeout, and again before the log tarball is created. Unlike ALRB's own copy, its removal therefore does not depend on ALRB exiting cleanly.

The payload script must not set the proxy. The payload command runs after the presetup, so an export X509_USER_PROXY=... in it would override the presetup. The pilot removes any such export from the command written to the container script.

Why the script must be sourced, not executed. An export in a script that runs as a separate process is lost when the process exits. Setting ALRB_CONT_PRESETUP to the proxy file itself makes ALRB try to execute the proxy:

/alrb/.bashrc: line 93: /srv/x509up_u25606_prod-payload.proxy: Permission denied

Fallback. If the presetup cannot be used, the pilot exports X509_USER_PROXY as before, so that a container is never left without a proxy. ALRB then copies the proxy, and a running payload does not see renewals. This happens in two cases:

• The site already sets ALRB_CONT_PRESETUP. The pilot does not override it and logs a warning.
• The presetup script or the temporary copy cannot be written.

Derived copies

These are not separate proxies, but they matter for which file a process actually reads.

Copy	Where	Updated on renewal?
ALRB's copy	ALRB's directory under ALRB_CONT_CHOME (the work directory)	No. Only made for payloads running with the pilot proxy, and when the fallback applies. Otherwise ALRB gets no proxy to copy.
<basename>-container-<label>.proxy	Work directory, while a stage-in/out or file-open container runs	No; the containers are short-lived. Made just before the container starts; removed when it finishes and before the log tarball.
<proxy>.tmp	Next to the job proxy during a renewal	Temporary; removed on success, on failure and before the log tarball.
Read-only file system fallback	PILOT_HOME	Written by the download code if the target directory is not writable; the download then redirects X509_USER_PROXY there. The job proxy renewal refuses such a download and undoes the redirect.

Cleanup and security

Before the log tarball is created, the pilot removes the job proxy from the work directory. This covers:

• the paths known from X509_UNIFIED_DISPATCH and the pilot cache;
• the names the pilot generates for both variants of the job proxy;
• any .tmp leftovers from a renewal;
• any temporary copy made for a stage-in/out or file-open container (*-container-*.proxy).

X509_UNIFIED_DISPATCH and the cached payload proxy path are then reset, so they cannot leak into the next job. Only files located directly in the work directory are removed.

As a second safety net, job proxy file names always end in .proxy, which matches the ATLAS redundant-file cleanup pattern *proxy.

The presetup scripts (pilot_proxy_presetup*.sh) contain only a path and may appear in the log tarball.

Since X509_USER_PROXY is unset in the container commands, ALRB no longer copies the pilot proxy into the work directory for stage-in/out, the file-open check or the debug command container. The only remaining case is the payload of jobs that run with the pilot proxy.

Configuration

Setting	Where	Default	Effect
-t, --noproxyverification	Pilot option	verification on	Disables all proxy handling for jobs: no job proxy is downloaded on any queue type, and no proxy is checked or renewed during the job.
-u, --verifypayloadproxy	Pilot option	verification on	Despite its name, the flag disables payload proxy verification, and therefore the job proxy on analysis queues.
proxy_verification_time	pilot/util/default.cfg, [Pilot]	600	Interval in seconds for the periodic checks of the pilot proxy and the job proxy.
payload_proxy_from_server	pilot/util/default.cfg, [Pilot]	True	Whether the job proxy is downloaded on analysis queues (not used on unified dispatch queues).
pandaserver	pilot/util/default.cfg, [Pilot]	https://pandaserver.cern.ch:25443	PanDA server used for all downloads (overridden by --url / --port).
ALRB_CONT_PRESETUP	Site environment	unset	If set by the site, the pilot does not set its own value and exports X509_USER_PROXY instead (see fallback).

The renewal timing of the job proxy is fixed in pilot/user/atlas/proxy.py and is not configurable:

Constant	Value	Meaning
JOB_PROXY_RENEWAL_THRESHOLD	24 h	Renew the job proxy below this remaining validity
JOB_PROXY_HARD_FLOOR	10 min	Fail the job (1391) below this if the proxy could not be renewed
JOB_PROXY_RENEWAL_BACKOFF	1 h	Delay after a renewal that did not solve the problem
JOB_PROXY_EXPECTED_LIFETIME	72 h	Minimum validity of a proxy served by the PanDA server (a shorter one is reported)
JOB_PROXY_LIFETIME_GRACE	20 min	Grace applied to the expected validity

pilot/user/atlas/default.cfg is an identical copy that the configuration loader does not read. If HARVESTER_PILOT_CONFIG is set, that file is read instead of pilot/util/default.cfg.

Error codes

Code	Name	Meaning in the proxy context
1163	NOPROXY	No valid grid proxy (pilot proxy).
1177	NOVOMSPROXY	Pilot proxy renewal failed during a job; the payload is killed.
1370	VOMSPROXYABOUTTOEXPIRE	Internal code, not a failure: tells the job monitor to renew the pilot proxy.
1373	CERTIFICATEHASEXPIRED	The proxy certificate has expired; the payload is killed.
1382	PROXYTOOSHORT	Pilot proxy has less than 72 h left before a job download; no new job is taken.
1391	PAYLOADPROXYDOWNLOADFAILURE	A job proxy could not be downloaded or verified at job start, or could not be renewed before it expired (payload killed).

Testing the renewal

The renewal can be tested with a short job by making the pilot believe that the job proxy expires soon.

Test settings:

File	Line	Test value	Release value
pilot/user/atlas/proxy.py	60	_TEST_JOB_PROXY_RENEWAL = True	_TEST_JOB_PROXY_RENEWAL = False
pilot/util/default.cfg	154	proxy_verification_time: 60	proxy_verification_time: 600

In test mode, the cached validity of the job proxy (never the pilot proxy) is clamped to the renewal threshold plus 5 minutes (86700 s) from each verification. A renewal is therefore due 5 minutes after each download, whatever the threshold. With the values above, this means about one renewal every 6 minutes, i.e. three renewals in a 20 minute job. The 72 hour check on downloaded proxies is skipped in test mode.

The unit tests in pilot/test/test_job_proxy_renewal.py (TestReleaseGuards) fail if either test value is left in place, so neither can be released by accident.

Expected pilot log, counting from the job proxy download (T). The examples are for an analysis queue; on a unified dispatch queue the messages say unified instead of payload.

1. At T: TEST MODE: clamping the validity of the payload proxy to 86700 s and caching the validity ends from arcproxy: cache['payload'] = [...].
2. About every 60 s: the payload proxy has 24.0xh left (renewal below 24h), without an arcproxy call.
3. Around T+6, T+12 and T+18 minutes:
   • the payload proxy is about to expire - downloading a new one for ...
   • the download to ...-payload.proxy.tmp
   • the voms-proxy-info dump
   • exactly one arcproxy call
   • renewed the payload proxy: ... (valid until ...)

Checking the containers. Also in the pilot log:

• The payload setup command starts with unset X509_USER_PROXY;export ALRB_CONT_PRESETUP="source /srv/pilot_proxy_presetup.sh";.
• For a containerised stage-in: copied ... to .../x509up_..._prod-container-stage-in.proxy for the stage-in container. The command contains unset X509_USER_PROXY;export ALRB_CONT_PRESETUP="source /srv/pilot_proxy_presetup_stage-in.sh";. Afterwards, removed container proxies from the work directory: [...].
• No *-container-*.proxy file remains in the work directory listing after stage-in.

A production job with containerised stage-in/out should also be run, since the container change affects all jobs.

Checking what the payload sees. Have the test payload print the proxy's inode once a minute:

for i in $(seq 1 20); do
  echo "$(date +%T) $X509_USER_PROXY inode=$(stat -c %i $X509_USER_PROXY) mtime=$(stat -c %y $X509_USER_PROXY | cut -c12-19) md5=$(md5sum < $X509_USER_PROXY | cut -c1-8)"
  sleep 60
done

The inode and mtime should change at each renewal, while X509_USER_PROXY stays /srv/.... Because of the server-side cache, the md5 may stay the same; the inode change still proves that the payload sees the renewed file.

The unified dispatch path, including the renewal before stage-out, needs a separate test on a unified dispatch queue.

Known limitations and open items

Pilot proxy (existing renewal mechanism):

• Payload containers that use the pilot proxy (production jobs, and user jobs without a job proxy) never see a renewed pilot proxy, since ALRB's copy is used and the proxy lives outside the work directory. For the same reason, ALRB still copies the pilot proxy into the work directory for these payloads. A user job running with the pilot proxy is reported with a warning (no job proxy available - the payload of this user job will use ...).
• The renewed proxy is written over the proxy in use before it has been verified.
• The renewal window is 30 minutes (40 minutes threshold, job killed on the first failed download).
• The cached check can mask an expiring certificate when the certificate ends before the VOMS attribute. For server-issued proxies both end times have been measured to be identical; for the pilot proxy provided by Harvester this has not been checked.
• A renewal of the pilot proxy that happens while the unified dispatch stage-out has switched X509_USER_PROXY would write a production proxy into the user proxy's path.

Very long jobs (up to 8 days):

• The pilot's maximum running time is taken from the job's maxwalltime (push mode), otherwise PQ.maxtime, otherwise --lifetime (default 324000 s = 90 h). Queues intended for such jobs need PQ.maxtime (or the job walltime) and the batch walltime set accordingly, or the job is failed with REACHEDMAXTIME (1213).
• Every such job depends on repeated renewals of every proxy it uses. For the job proxy this is in place: renewal 24 hours before expiry, and a renewed proxy is only used if it is valid longer than the current one. The pilot proxy still uses the mechanism described above (40 minute threshold, job killed on the first failure). The same early renewal, verification before replacement and retries are planned for it, and are required before such jobs go into production.

Code structure:

• The two variants of the job proxy are implemented by separate code paths, with separate download functions, state (pilot cache vs X509_UNIFIED_DISPATCH) and file names, although the credential is the same. Consolidating them into one download path, one state variable and one file name, with the queue type deciding which operations use the job proxy, is planned.

Containers:

• The unset-plus-presetup handling for the stage-in/out and file-open containers follows the ALRB maintainers' recommendation. It has so far only been validated in unit tests; validation on a site is pending. The presetup itself has been validated for the payload container on CERN-PTEST.
• On sites that set their own ALRB_CONT_PRESETUP, the fallback applies: ALRB copies the proxy, including the pilot proxy for stage-in/out.
• The other experiment plugins (generic, epic, ska, darkside) still export X509_USER_PROXY in their container commands.

Other:

• On sites without arcproxy, the job proxy cannot be verified and is therefore not renewed (the same applies to the pilot proxy).
• Token-based storage access (e.g. a Rucio client requesting tokens) is not covered yet. If the token request is authenticated with the X.509 proxy, the mechanism above still applies. Direct I/O with token-only storage would need a token inside the payload container, renewed in the same way.