Repository navigation
SK-3075: flowvault public release (Detect V2 + new interfaces) - #292
Merged
Merged
Conversation
…e columns - upload_files: upload per-column files concurrently (bounded pool of 5) and stream each file from disk, so large/many-file records finish inside the signed-URL window with flat memory - client: dedicated no-retry httpx client for signed-URL PUTs so a streamed body is never replayed on a 5xx - validations: reject duplicate column names within an upload/delete record - tests: updated for streaming/parallel path, added duplicate-column + multi-column coverage Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- UploadFilesRequestColumn: remove content_type (it never reached the /v2/files/upload request; the PUT Content-Type is inferred from the file name). Updated contract baseline, README, and tests. - README: correct the upload note (concurrent + streamed, not sequential) - get(): remove dead error_count assignments Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* SK-3075: add Detect V2 to flowvault and skyvault SDKs
Implement Detect V2 (deidentify_string, deidentify_file, get_run,
reidentify_string, reidentify_file, check_guardrails) in the shared
`common` package and re-export it from both flowvault and skyvault.
common:
- detect models, controller, request/response mappers, run polling,
file-output writer, and validations under common/vault/detect
- V2 detect enums under common/utils/enums/detect.py; V2 detect
messages in common SkyflowMessages
- behaviour ported from the Java flowvault SDK @ flowvault-release/26.10.1
(exponential poll backoff 1..32, submit-only status None, minimal
client-side validation, auto-write of processed files)
flowvault:
- wire DetectController via make_skyflow_class(detect_cls=...)
- build the common detect REST client on VaultClient and expose
get_strings_v2_api / get_files_v2_api / get_guardrails_v2_api
- re-export skyflow.vault.detect and the V2 enums
- detect_api samples, README section, and unit tests
skyvault:
- replace the removed V1 detect API with V2 (same method/class names)
- migrate the hand-written layer to the regenerated client (drop V1
prefix, Skyflow -> SkyflowAuth) and update tests
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* SK-3075: fix generated version fallback and refresh API contract baselines
- common/skyvault generated rest version.py: guard metadata.version() with
try/except PackageNotFoundError (matches flowvault), so importing the
package never fails when the distribution metadata is absent.
- Refresh flowvault/skyvault public-API contract baselines for the Detect V2
surface (flowvault adds the V2 enums; skyvault replaces V1 detect with V2).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* SK-3075: make common detect tests unittest-compatible
The common package CI runs tests with `python -m unittest` (no pytest
installed), so rewrite common/tests/vault/detect/test_detect.py as a
unittest.TestCase (self.assertRaises, tempfile instead of the pytest
tmp_path fixture). Runs green under both unittest and pytest.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* SK-3075: split detect DataSourceType into per-SDK DeidentifySourceType and ReidentifySourceType
Mirror the Java Detect V2 contract change: the single DataSourceType enum is split
into DeidentifySourceType (deidentify_file data_source and the output_type of
DeidentifyFileResponse / GetRunResponse) and ReidentifySourceType (reidentify_file
data_source and ReidentifyFileResponse output_type).
Like Java, the two source enums are defined per SDK rather than in common:
- skyvault DeidentifySourceType: BASE64, SKYFLOW_ID, PRESIGNED_URL
- flowvault DeidentifySourceType: BASE64, PRESIGNED_URL (no skyflow id)
- ReidentifySourceType: BASE64, PRESIGNED_URL in both (re-identification never
accepts a skyflow id)
common stays SDK-agnostic: the controller/mappers resolve the response output_type
through the per-SDK VaultClient (get_deidentify_source_type / get_reidentify_source_type),
and the file-output writer compares the output type by value. Updates both SDK enum
re-exports, the reidentify_file sample, and the unit/re-export tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* SK-3075: rename the Detect README section heading to "Detect"
"# DetectController — Detect V2" produced an awkward anchor
(#detectcontroller--detect-v2) and didn't match the cross-SDK convention; the
Go flowvault README uses "Detect". Renamed to "# Detect" (anchor #detect); the
section intro already states it is Detect V2 and how to get the controller.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* SK-3076 add v1-deprecation warning (#288)
* SK-3076 add v1-deprecation warning
* SK-3076 fix ci-check
* SK-3075: fix auth, version, and skyvault data-plane after codegen regen
The 865a4f5 regeneration renamed generated REST client methods but the
consuming code was not fully updated, breaking bearer-token auth (every
e2e failed with "Failed to generate bearer token") and the skyvault
data-plane:
- common/service_account/_utils.py: call get_bearer_token() (renamed from
authentication_service_get_auth_token).
- common/utils/_skyflow_messages.py: resolve the host SDK version via an
importlib.metadata fallback so messages/metrics report the real version
instead of 0.0.0.
- skyvault/skyflow/service_account/__init__.py: re-export from
common.service_account (skyvault's generated authentication client was
removed by the regen), matching flowvault.
- skyvault/skyflow/vault/controller/_vault.py: rewire record/token/query
calls to the new generated method names; move file upload to the files
client (upload_file_v2).
- skyvault/skyflow/vault/client/client.py: add get_files_api().
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* SK-3075: refresh API contract baselines for DeidentifySourceType/ReidentifySourceType split
The detect DataSourceType enum was split into per-SDK DeidentifySourceType
and ReidentifySourceType, but the committed griffe baselines were not
regenerated, so Contract Tests failed on public API drift. Regenerate both
skyvault and flowvault baselines to record the intended enum change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* SK-3075: update skyvault unit tests for the new-codegen data-plane and common auth
Mirror the controller/auth migration in the unit tests so they exercise the
methods the code now calls:
- test__vault.py: rename the mocked generated methods to the new names
(batch_operation/insert_records/update_record/delete_records/get_records/
execute_query/detokenize/tokenize), point file-upload at the files client
(get_files_api + upload_file_v2), and drop byot from the batch_operation
expectation (batch_operation no longer takes it; BYOT rides on per-record
tokens).
- service_account/test__utils.py and utils/test__utils.py: the public
skyflow.service_account now routes through common, so patch
common.service_account._utils.* and assert get_bearer_token on the mocked
auth client instead of the removed authentication_service_get_auth_token.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* SK-3075: split detect RedactionLevel into a per-SDK rule (match Java)
Mirror skyflow-java's SK-3175 change. RedactionLevel was one common class
carrying all four wire fields for every vault type. Replace it with an
abstract RedactionRule base in common (exposing the four wire accessors)
and a per-SDK RedactionLevel:
- skyvault RedactionLevel carries entity_name + redaction_type
- flowvault RedactionLevel carries token_group_name + redaction_pattern
The reidentify mapper reads the wire fields through the rule's accessors,
so each vault type only sends the fields its API supports. Refreshed the
skyvault API contract baseline for the new class.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* SK-3075: fix common detect test for the RedactionLevel split
common.vault.detect no longer exports a concrete RedactionLevel (it exports
the RedactionRule base), so the reidentify mapping test uses a small concrete
RedactionRule double instead of importing RedactionLevel.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* SK-3075: make the skyvault sdist self-contained
The skyvault sdist omitted common/, so rebuilding a wheel from it (the release
smoke test) failed looking for a sibling common/ that only exists in the repo
checkout. Mirror flowvault: vendor common/ into the sdist via a CustomSdist
command plus graft common, and fall back to the vendored copy when the sibling
checkout is absent, so sdist-to-wheel builds and installs cleanly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* SK-3075: regenerate skyvault files client for multipart file upload
UploadFileRequestV2.file now carries format:binary in the spec, so the
regenerated upload_file_v2 sends file as a multipart part
(files={"file": file}) instead of a string form field, fixing the 500
"Expected primitive type, got <class 'list'>" from data-plane file upload.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* SK-3075: restore working skyvault generated version.py
The files-client regen also rewrote generated/rest/version.py to look up
metadata.version("skyflow.generated.rest"), a distribution that is never
installed (the module ships inside the "skyflow" dist), so it raised
PackageNotFoundError. unittest discover enumerates the package __all__
(which includes __version__), tripping the lazy import and failing
test/run-tests. Restore the metadata.version("skyflow") lookup with the
0.0.0 fallback. Durable fix belongs in the fern generator config.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* SK-3075: re-trigger PR CI on the restored generated version.py
No code change (empty commit). The previous push updated the PR head and
the release branch to the same SHA in one step, so GitHub created only the
push-triggered release run and never fired the pull_request PR CI. This
empty commit fires pull_request:synchronize so test/run-tests re-runs on
the restored version.py. The published 2.1.3.dev0+deaa724 wheel is built
from the identical code and stays valid.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* SK-3284: add detect V1 to V2 migration guide for skyvault python sdk
Mirrors the Java migration doc as a repo reference: method and class
mapping, request field mapping, per-method V1/V2 examples, file
behaviour differences, and a migration checklist. Confirmed against the
V2 detect implementation and the V1 API on main.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* SK-3075: document Detect V2 in skyvault and flowvault READMEs and samples
Rewrite the skyvault README Detect section from the removed V1 API to V2
(method table, configuration contract, Detect configuration reference,
file run status and output handling) and replace the V1 detect samples
with V2 equivalents. Mirror the same V2 Detect structure into the
flowvault README with flowvault semantics (token-group destination,
token_group_name / redaction_pattern) and enhance the flowvault samples
to cover inline configuration, submit-only / poll / presigned, and the
failure branches.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* SK-3075: align flowvault Detect V2 with Java and add detect to flowvault API contract
Cross-SDK parity review flagged four Detect V2 gaps in the Python flowvault
path versus the Java reference. All fixes live in shared common/ code and so
apply to both vaults.
- deidentify_file source rule now counts exactly one of file_path, file, value;
a missing source and multiple sources raise distinct errors (adds the
missing-source message).
- Shared deidentify/reidentify-file validation messages are vault-neutral and no
longer name SKYFLOW_ID, which flowvault does not offer.
- With neither configuration_id nor configuration set, a configuration carrying
only the vault id is sent, matching the Java behaviour.
- processed_file_extension resolves case-insensitively.
Also add skyflow.vault.detect to the flowvault public API contract and
regenerate both api-report baselines; skyvault is unchanged, flowvault gains the
detect surface.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* SK-3075: expose includePrefix / prefixOverride on Detect Entity and CustomType
The generated wire types already carry these fields; surface them on the public
Detect config so callers can control VAULT_TOKEN tokenization, matching the Java
design doc.
- Add the IncludePrefix enum (YES / NO) and re-export it from both vaults'
skyflow.utils.enums.
- Entity gains include_prefix and prefix_override.
- CustomType gains deidentification_type, destination, include_prefix and
prefix_override.
- The request mapper sends all of them when set; omitted fields fall through to
the server defaults.
- Regenerate both api-report baselines (the new public surface is the
IncludePrefix enum; Entity / CustomType are re-export aliases).
* SK-3289: address Detect docs and sample parity feedback (Python) Fix the Detect README and sample gaps raised in the final Detect feedback, for both the flowvault and skyvault Python SDKs. flowvault README: - drop the unsupported SKYFLOW_ID file source (the enum is BASE64/PRESIGNED_URL only) - document Entity/CustomType include_prefix and prefix_override, and CustomType deidentification_type and destination - add where-to-create pointers for the configuration id, token groups and redaction patterns - add response examples and a common-errors section - make the re-identify example a round trip; branch re-identify file on output_type skyvault README: - make the re-identify example a round trip that stores tokens in a table.column; branch re-identify file on output_type - add a bold warning that deidentify_file no longer waits by default - note the server's default toxicity check; fix the stale v1 EOL banner samples (both SDKs): print request_id in Detect error handlers. Nomenclature: drop the "Detect V2" label from both READMEs and the samples README; rename the migration guide to docs/migrate_detect.md, retitle it, cite skyflow 3.0.0, and promote the deidentify_file behaviour change. setup.py version is managed by release automation and left unchanged; the guide now names 3.0.0 as the target. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * SK-3289: fix internal parity-gap issues in Python SDK flowvault: - Disable the generated client's hidden retry layer so the SDK's configured max_retries (default 0) is honored and POSTs are not silently auto-retried. - Pass the bearer token as None when absent so no empty "Authorization: Bearer " header is sent. - Build insert/get wire objects inside the try block so pydantic ValidationError surfaces as a SkyflowError. - Rewind a caller-supplied file_object to the start before upload (without closing the caller's handle). - Guard client re-init and close with a lock. common (shared by flowvault and skyvault): - Validate env with isinstance instead of enum membership to avoid a TypeError on Python 3.9-3.11 for non-Env values. - Attach the skyflow-python log handler once instead of clearing and re-adding it on every log call. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * SK-3289: split file data format into deidentify and reidentify (Python) Mirror the Java flowvault change: reidentify_file only supports the file formats that carry VAULT_TOKEN, so the single FileDataFormat enum is split into separate types. - enums: rename FileDataFormat -> DeidentifyFileDataFormat, add ReidentifyFileDataFormat (csv, docx, json, jsonl, pptx, txt, xlsx, xml) - models: FileOutput uses DeidentifyFileDataFormat; add ReidentifyFileOutput; ReidentifyFileResponse.output is now a list of ReidentifyFileOutput - mappers: separate deidentify/reidentify format resolvers; an unknown reidentify format maps to None - re-exports, samples, READMEs and tests updated across flowvault and skyvault - regenerate the public API contract baselines
Promptless documentation updates
|
…10.0 # Conflicts: # README.md
|
🔐 Gitleaks Findings: 4 issue(s) detected 🔸 Rule: 🔸 Rule: 🔸 Rule: 🔸 Rule: |
Public API contract change (
|
Public API contract change (
|
saileshwar-skyflow
changed the title
The generated REST clients embed spec-derived example token values in docstrings (not real secrets). Add common/*/generated paths to the gitleaks allowlist so the PR scan stops flagging them. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
✅ Gitleaks Findings: No secrets detected. Safe to proceed! |
|
✅ Gitleaks Findings: No secrets detected. Safe to proceed! |
skyflow-himanshupal
approved these changes
Nov 11, 2026
skyflow-bharti
approved these changes
Nov 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Promotes the
flowvault-release/26.10.0work tomainfor the publicskyflow-flowvault-pythonrelease.Included
common/and re-exported from both SDKs, with the file-data-format split:DeidentifyFileDataFormat/ReidentifyFileDataFormat+ReidentifyFileOutput(matches the Java reference).query,get_tokens,upload_files,delete_files; parallelized/streamed file uploads; duplicate-column rejection.*/api-report/skyflow.api.json) regenerated for both modules.Notes
flowvault/setup.pycarries an internal dev version (1.0.1.dev0+…); the published version is set by the release tag (flowvault/v<semver>), not setup.py, so this is harmless on merge.main(verified).🤖 Generated with Claude Code