Skip to content

Stable 26.3 Backport of #131963 - Clear --jwt and the other credential options from the command line of clickhouse-client and clickhouse-keeper-client - #2931

Draft
blau-ai wants to merge 2 commits into
stable-26.3from
backports/stable-26.3/131963
Draft

blau-ai wants to merge 2 commits into
stable-26.3from
backports/stable-26.3/131963

Conversation

@blau-ai

@blau-ai blau-ai commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator

Changelog category (leave one):

  • Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

clickhouse-client now clears the values of --jwt, --ssh-key-passphrase, --one-time-password and --oauth-client-secret from its command line, as it already did for --password, so they are no longer visible in ps and /proc/<pid>/cmdline. clickhouse-keeper-client now does the same for --password and --identity. Closes ClickHouse#131191. (ClickHouse#131963 by @groeneai)

Documentation entry for user-facing changes

...

CI/CD Options

Exclude tests:

  • Fast test
  • Integration Tests
  • Stateless tests
  • Stateful tests
  • Unit tests
  • Performance tests
  • Aarch64 tests
  • All with ASAN
  • All with TSAN
  • All with MSAN
  • All with UBSAN
  • All with Coverage
  • All Regression
  • Disable CI Cache

Regression jobs to run:

  • Fast suites (mostly <1h)
  • Aggregate Functions (2h)
  • Alter (1.5h)
  • Benchmark (30m)
  • CAS (content-addressed storage; Antalya only)
  • ClickHouse Keeper (1h)
  • Iceberg (2h)
  • LDAP (1h)
  • OAuth (5m)
  • Parquet (1.5h)
  • RBAC (1.5h)
  • SSL Server (1h)
  • S3 (2h)
  • S3 Export (2h)
  • Swarms (30m)
  • Tiered Storage (2h)

…ptions-from-command-line

Clear `--jwt` and the other credential options from the command line of `clickhouse-client` and `clickhouse-keeper-client`
@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Workflow [PR], commit [7254e52]

@zvonand

zvonand commented Nov 20, 2026

Copy link
Copy Markdown
Member

@blau-ai

@blau-ai

blau-ai commented Nov 20, 2026

Copy link
Copy Markdown
Collaborator Author

CI triage for 9b67254

Summary: 3 distinct problems. 1 is caused by this PR, 1 is infra, and 1 is the DCO process check.

Check(s) Cause Caused by this PR?
Stateless tests: amd_debug, parallel, amd_ubsan, parallel, amd_asan, distributed plan, parallel 4/4, amd_debug, distributed plan, s3 storage, parallel The new test 05813_clear_credentials_from_command_line uses a client option that does not exist on 26.3 ✅ Yes
Integration tests: amd_asan … 1,2,3,4,5,7,8/8 and amd_binary 1,2,4,5/5 Docker image pull timed out in cluster.start() ❌ No (infra)
DCO The cherry-picked merge commit has no Signed-off-by ❌ Not a code issue

1. 05813_clear_credentials_from_command_line: caused by this PR

It is the only failing stateless test, and it fails the same way in all 4 jobs:

05813_clear_credentials_from_command_line.sh: line 59: /proc/2166538/cmdline: No such file or directory
... (x5)
stdout:
client, separate values
client, values after =
client, values after spaced =
client, --password without a value
keeper-client
keeper-client, values after :
--password:
--identity:

Root cause: each of the 4 clickhouse-client cases passes --oauth-client-secret. That option does not exist on stable-26.3. programs/client/Client.cpp here only defines --login, --oauth-url, --oauth-client-id and --oauth-audience. So the client exits right away with an "unrecognized option" error. It never connects to the listener, and /proc/<pid>/cmdline is gone by the time the test reads it. Upstream has this option, so the test passes there.

The first keeper-client case also fails, but only as a side effect. The test counts expected connections, and the 4 client cases that never connected throw that count off. The last keeper-client case still gives the expected output. The C++ change itself looks correct: the case that did run zeroes the values as expected.

Suggested fix (test only, no code change needed):

  • 05813_clear_credentials_from_command_line.sh: remove --oauth-client-secret SECRET, --oauth-client-secret=SECRET and --oauth-client-secret = SECRET from the three client invocations that pass it (lines 67–76).
  • 05813_clear_credentials_from_command_line.reference: remove the three --oauth-client-secret / --oauth-client-secret= lines.
  • "--oauth-client-secret"sv can stay in secret_options in clearPasswordFromCommandLine.cpp. It does nothing on 26.3 and keeps the code the same as upstream. Dropping it would also be fine.

I can push this to the PR branch if you want it.

2. Integration tests: infra, not caused by this PR

Across the 11 failing integration shards there are thousands of FAILs in unrelated modules: test_database_replicated_settings (780), test_storage_delta, test_storage_s3, test_grpc_protocol, test_ttl_move, and others. I checked every FAIL entry in the report, and every one has the same traceback, in the docker pull step of cluster setup:

File: helpers/cluster.py:3400 - in start
    retry(log_function=logging_pulling_images, retries=3, delay=8, jitter=8)(run_and_check, images_pull_cmd, timeout=600)
...
E   Failed: Timeout (>900.0s) from pytest-timeout.

No test got as far as running a query. The PR changes nothing that integration tests use: it touches clearPasswordFromCommandLine and keeper-client main, not the server. Next step: re-run these shards once the image registry or runners are healthy.

3. DCO: process check only

The only commit is the cherry-picked upstream merge commit 9b67254817 ("Merge pull request ClickHouse#131963 …"), and it has no Signed-off-by line. This is the usual state for automated backports and says nothing about the code. If it blocks merging, a maintainer can override or sign it off. Rewriting the commit would need a force-push, which I won't do.


Sources: praktika report PRs/2931/9b6725481750cfc5b4cf421696a378ffcf94c311/result_pr.json and the stable-26.3 sources. I did not re-run anything locally, because the report and the source code settled every question here.

@zvonand

zvonand commented Nov 20, 2026

Copy link
Copy Markdown
Member

@blau-ai fix fails caused by this PR here

…mand_line

clickhouse-client of 26.3 has no --oauth-client-secret option, so it exited
with UNRECOGNIZED_ARGUMENTS before the test could inspect /proc/<pid>/cmdline.
clearPasswordFromCommandLine still lists the option, which is harmless.

Signed-off-by: blau-ai <blau-ai@users.noreply.github.com>
@blau-ai

blau-ai commented Nov 20, 2026

Copy link
Copy Markdown
Collaborator Author

Fixed in 7254e52 (pushed to backports/stable-26.3/131963).

Failure this PR caused

05813_clear_credentials_from_command_line failed in 4 stateless jobs: amd_debug parallel, amd_ubsan parallel, amd_asan distributed plan 4/4, and amd_debug distributed plan s3.

  • Root cause: the test was backported from upstream unchanged, and it passes --oauth-client-secret to clickhouse-client. That option does not exist in 26.3's client, so the client exited right away with Code: 552. DB::Exception: Unrecognized option '--oauth-client-secret=SECRET'. (UNRECOGNIZED_ARGUMENTS). The test then found no /proc/<pid>/cmdline to read. Its connection counting also went out of step after that, which is why the run hung for about 9 minutes.
  • Fix: I removed --oauth-client-secret from the three clickhouse-client invocations and from the .reference. The C++ change is untouched. clearPasswordFromCommandLine still lists that option, which does no harm.
  • Validation (blau-ch test, run against this PR's CI binaries, no build):
    • Update README.md #1, reason "Diagnose why 05813_clear_credentials_from_command_line fails on stable-26.3 CI binary", amd_release binary with a temporary stderr dump in the test: reproduced the UNRECOGNIZED_ARGUMENTS error above.
    • Update README.md #2, reason "Verify fixed 05813_clear_credentials_from_command_line (no --oauth-client-secret) passes on PR debug binary", amd_debug binary: 05813_clear_credentials_from_command_line: [ OK ] 1.33 sec.

Failures not caused by this PR

  • Integration tests (amd_binary and amd_asan shards): all 2631 failed test cases fail in cluster.start() while pulling docker images (helpers/cluster.py:3400 ... images_pull_cmd). That is a CI infrastructure problem, unrelated to this change. A rerun should clear them.
  • DCO: the original backport commit 9b67254 has no Signed-off-by line. My commit is signed off. Fixing the old commit would mean rewriting history (force-push), which I'm not allowed to do, so a maintainer needs to handle it: override DCO or accept it.

CI on the new head still has to confirm the fix. My local run was a single run against the debug CI binary, without sanitizers.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants