# Anonymize Functions Source: https://docs.rehydra.ai/api-reference/anonymize Functions for anonymizing text Rehydra provides several functions for anonymizing text, from simple one-off calls to reusable instances. ## anonymizer.anonymize() The primary method on an `Anonymizer` instance. ### Signature ```typescript theme={null} anonymize( text: string, locale?: string, policy?: Partial ): Promise ``` ### Parameters The text to anonymize Locale hint (e.g., `'de-DE'`, `'en-US'`). Helps with semantic enrichment. Policy override for this call. Merged with default policy. ### Returns ```typescript theme={null} interface AnonymizationResult { anonymizedText: string; entities: Array<{ type: PIIType; id: number; start: number; end: number; confidence: number; source: 'REGEX' | 'NER' | 'HYBRID'; semantic?: { gender?: 'male' | 'female' | 'neutral' | 'unknown'; scope?: 'city' | 'country' | 'region' | 'unknown'; title?: string; }; }>; piiMap?: { // undefined when mode is 'anonymize' ciphertext: string; iv: string; authTag: string; }; stats: { countsByType: Record; totalEntities: number; processingTimeMs: number; modelVersion: string; leakScanPassed?: boolean; }; } ``` ### Example ```typescript theme={null} const anonymizer = createAnonymizer({ ner: { mode: 'quantized' } }); await anonymizer.initialize(); const result = await anonymizer.anonymize( 'Contact John at john@example.com', 'en-US' ); console.log(result.anonymizedText); // "Contact at " console.log(result.entities); // [ // { type: 'PERSON', id: 1, start: 8, end: 12, confidence: 0.95, source: 'NER' }, // { type: 'EMAIL', id: 1, start: 16, end: 32, confidence: 1.0, source: 'REGEX' } // ] console.log(result.stats); // { totalEntities: 2, processingTimeMs: 150, ... } ``` *** ## anonymize() Standalone function for one-off anonymization (regex-only by default). ### Signature ```typescript theme={null} function anonymize( text: string, locale?: string, policy?: Partial ): Promise ``` ### Example ```typescript theme={null} import { anonymize } from 'rehydra'; const result = await anonymize('Email: test@example.com'); console.log(result.anonymizedText); // "Email: " ``` Creates a temporary anonymizer for each call. For multiple texts, use `createAnonymizer()` instead. *** ## anonymizeRegexOnly() Fast regex-only anonymization without NER. ### Signature ```typescript theme={null} function anonymizeRegexOnly( text: string, policy?: Partial ): Promise ``` ### Example ```typescript theme={null} import { anonymizeRegexOnly } from 'rehydra'; const result = await anonymizeRegexOnly( 'Card: 4111111111111111, IBAN: DE89370400440532013000' ); console.log(result.anonymizedText); // "Card: , IBAN: " ``` ### Performance | Function | Time (2K chars) | | ---------------------- | --------------- | | `anonymizeRegexOnly()` | \~5-10 ms | | `anonymize()` with NER | \~150-200 ms | *** ## anonymizeWithNER() One-off anonymization with NER model. ### Signature ```typescript theme={null} function anonymizeWithNER( text: string, nerConfig: Omit & { mode?: 'standard' | 'quantized' }, policy?: Partial ): Promise ``` ### Example ```typescript theme={null} import { anonymizeWithNER } from 'rehydra'; const result = await anonymizeWithNER( 'Hello John Smith from Acme Corp!', { mode: 'quantized', onStatus: console.log } ); console.log(result.anonymizedText); // "Hello from !" ``` Downloads and loads the NER model on each call. For multiple texts, use `createAnonymizer()` instead. *** ## Policy Overrides Override default policy for specific calls: ```typescript theme={null} import { createAnonymizer, PIIType } from 'rehydra'; const anonymizer = createAnonymizer({ ner: { mode: 'quantized' } }); await anonymizer.initialize(); // Default behavior await anonymizer.anonymize('Email: test@example.com'); // Override for this call only await anonymizer.anonymize('Email: test@example.com', undefined, { enabledTypes: new Set([PIIType.PHONE]), // Only detect phones allowlistTerms: new Set(['test@example.com']), // Ignore this email }); // Keep countries visible (requires semantic: { enabled: true }) await anonymizer.anonymize('Meeting in Berlin, Germany.', undefined, { excludeLocationScopes: new Set(['country', 'region']), }); ``` *** ## Handling Results ### Access Anonymized Text ```typescript theme={null} const result = await anonymizer.anonymize(text); console.log(result.anonymizedText); ``` ### Check Detection Stats ```typescript theme={null} if (result.stats.totalEntities === 0) { console.log('No PII detected'); } console.log(`Found ${result.stats.countsByType['EMAIL'] || 0} emails`); ``` ### Validate Output ```typescript theme={null} if (!result.stats.leakScanPassed) { console.warn('Potential PII leak detected in output'); } ``` ## Related * [createAnonymizer](/api-reference/create-anonymizer) - Create an anonymizer instance * [rehydrate](/api-reference/rehydrate) - Restore original values * [Sessions](/api-reference/sessions) - Session-based anonymization # createAnonymizer Source: https://docs.rehydra.ai/api-reference/create-anonymizer Create a reusable anonymizer instance Creates and configures an anonymizer instance for PII detection and replacement. ## Signature ```typescript theme={null} function createAnonymizer(config?: AnonymizerConfig): Anonymizer ``` ## Parameters ### config (optional) Anonymization mode: * `'pseudonymize'` — Reversible. Returns an encrypted `piiMap` for later rehydration. * `'anonymize'` — Irreversible. No `piiMap` is returned; PII is permanently removed. NER model configuration Model mode: `'disabled'` | `'quantized'` | `'standard'` | `'custom'` Custom model path (required when mode is `'custom'`) Custom vocab path (required when mode is `'custom'`) Auto-download model if not present Confidence thresholds per PII type (0.0-1.0) Status message callback Download progress callback Enable case-insensitive fallback for detecting lowercase names. Runs a second NER pass on title-cased text and merges new detections. Doubles NER inference time but catches names like `"tom"` that the case-sensitive model would otherwise miss. Confidence penalty multiplier for case-fallback detections (0.0–1.0). Applied as `confidence * caseFallbackPenalty`. Lower values are stricter. Inference backend: `'local'` (ONNX on device) or `'inference-server'` (remote GPU server) URL of the inference server (required when backend is `'inference-server'`) Inference server request timeout in milliseconds Semantic enrichment configuration Enable semantic enrichment (gender/scope attributes) Auto-download semantic data if not present Status message callback Download progress callback Encryption key provider. If not provided, generates a random key. Storage provider for session-based PII map persistence Default anonymization policy Secrets/credentials detection configuration Enable secrets detection (API keys, tokens, connection strings, etc.) Paths to `.env` files for literal value redaction Redact secret values found in `.env` files Additional patterns to match secret key names Minimum value length to consider a secret Tag format for PII placeholder tags. Controls the delimiters and keyword. Opening delimiter (e.g., `"<"` for XML, `"[["` for bracket style) Closing delimiter (e.g., `"/>"` for XML self-closing, `"]]"` for bracket style) Tag keyword. Allows using e.g., `"REDACTED"` instead of `"PII"` Callback for validation warnings. When not provided, warnings are silently ignored — check `stats.leakScanPassed` in the result instead. Custom recognizer registry (uses default if not provided) ## Returns An `Anonymizer` instance with these methods: | Method | Description | | ----------------------------------- | ---------------------------------------- | | `initialize()` | Initialize the anonymizer (loads models) | | `anonymize(text, locale?, policy?)` | Anonymize text | | `session(sessionId)` | Create a session for persistent storage | | `dispose()` | Release resources | | `getRegistry()` | Get the recognizer registry | | `getNERModel()` | Get the NER model instance | | `isInitialized` | Check if initialized | ## Examples ### Basic (Regex Only) ```typescript theme={null} import { createAnonymizer } from 'rehydra'; const anonymizer = createAnonymizer(); await anonymizer.initialize(); const result = await anonymizer.anonymize('Contact: test@example.com'); console.log(result.anonymizedText); // "Contact: " await anonymizer.dispose(); ``` ### With NER ```typescript theme={null} const anonymizer = createAnonymizer({ ner: { mode: 'quantized', onStatus: (status) => console.log(status) } }); await anonymizer.initialize(); const result = await anonymizer.anonymize('Hello John Smith!'); // "Hello !" ``` ### With Semantic Enrichment ```typescript theme={null} const anonymizer = createAnonymizer({ ner: { mode: 'quantized' }, semantic: { enabled: true } }); await anonymizer.initialize(); const result = await anonymizer.anonymize('Contact Maria in Berlin'); // "Contact in " ``` ### With Storage ```typescript theme={null} import { createAnonymizer, InMemoryKeyProvider, SQLitePIIStorageProvider } from 'rehydra'; const storage = new SQLitePIIStorageProvider('./pii.db'); await storage.initialize(); const anonymizer = createAnonymizer({ ner: { mode: 'quantized' }, keyProvider: new InMemoryKeyProvider(), piiStorageProvider: storage, }); await anonymizer.initialize(); // Use sessions for automatic persistence const session = anonymizer.session('chat-123'); await session.anonymize('Hello John!'); ``` ### Irreversible Anonymization ```typescript theme={null} const anonymizer = createAnonymizer({ mode: 'anonymize', // No piiMap returned, PII permanently removed ner: { mode: 'quantized' }, }); await anonymizer.initialize(); const result = await anonymizer.anonymize('Contact John Smith at john@acme.com'); console.log(result.anonymizedText); // "Contact at " console.log(result.piiMap); // undefined ``` ### With Inference Server ```typescript theme={null} const anonymizer = createAnonymizer({ ner: { mode: 'quantized', backend: 'inference-server', inferenceServerUrl: 'http://gpu-server:8000/predict', } }); ``` ### With Custom Tag Format ```typescript theme={null} const anonymizer = createAnonymizer({ tagFormat: { open: '[[', close: ']]' } }); await anonymizer.initialize(); const result = await anonymizer.anonymize('Contact: test@example.com'); console.log(result.anonymizedText); // "Contact: [[PII type="EMAIL" id="1"]]" ``` ### With Custom Thresholds ```typescript theme={null} import { createAnonymizer, PIIType } from 'rehydra'; const anonymizer = createAnonymizer({ ner: { mode: 'quantized', thresholds: { [PIIType.PERSON]: 0.8, [PIIType.ORG]: 0.7, } } }); ``` ### With Custom Policy ```typescript theme={null} import { createAnonymizer, PIIType, createDefaultPolicy } from 'rehydra'; const policy = createDefaultPolicy(); policy.enabledTypes = new Set([PIIType.EMAIL, PIIType.PHONE, PIIType.PERSON]); policy.allowlistTerms = new Set(['Support Team', 'Help Desk']); policy.enableLeakScan = true; const anonymizer = createAnonymizer({ ner: { mode: 'quantized' }, defaultPolicy: policy, }); ``` ### Excluding Countries and Regions ```typescript theme={null} const anonymizer = createAnonymizer({ ner: { mode: 'quantized' }, semantic: { enabled: true }, defaultPolicy: { excludeLocationScopes: new Set(['country', 'region']), }, }); await anonymizer.initialize(); const result = await anonymizer.anonymize('Meeting in Berlin, Germany.'); // "Meeting in , Germany." ``` ## Related * [anonymize](/api-reference/anonymize) - The anonymize method * [Sessions](/api-reference/sessions) - Session-based persistence * [Storage Providers](/api-reference/storage-providers) - Available storage options # Crypto Utilities Source: https://docs.rehydra.ai/api-reference/crypto Encryption and key management functions Rehydra uses AES-256-GCM encryption via the Web Crypto API for securing PII maps. ## Key Providers ### InMemoryKeyProvider Generates a random key in memory. Key is lost when the process ends. ```typescript theme={null} import { InMemoryKeyProvider } from 'rehydra'; const keyProvider = new InMemoryKeyProvider(); const key = await keyProvider.getKey(); // Uint8Array (32 bytes) ``` Only for development/testing. The key is not persisted. ### ConfigKeyProvider Uses a pre-configured key from your environment. ```typescript theme={null} import { ConfigKeyProvider } from 'rehydra'; // Generate key: openssl rand -base64 32 const keyBase64 = process.env.PII_ENCRYPTION_KEY; const keyProvider = new ConfigKeyProvider(keyBase64); ``` ### KeyProvider Interface ```typescript theme={null} interface KeyProvider { getKey(): Promise; } ``` Implement for custom key storage: ```typescript theme={null} class VaultKeyProvider implements KeyProvider { async getKey(): Promise { const secret = await vault.getSecret('pii-encryption-key'); return base64ToUint8Array(secret); } } ``` *** ## Encryption Functions ### encryptPIIMap() Encrypts a PII map. ```typescript theme={null} import { encryptPIIMap } from 'rehydra'; const piiMap = new Map([ ['EMAIL:1', 'john@example.com'], ['PERSON:1', 'John Smith'], ]); const encrypted = await encryptPIIMap(piiMap, key); // { ciphertext: '...', iv: '...', authTag: '...' } ``` ### decryptPIIMap() Decrypts a PII map. ```typescript theme={null} import { decryptPIIMap } from 'rehydra'; const piiMap = await decryptPIIMap(encrypted, key); // Map { 'EMAIL:1' => 'john@example.com', ... } ``` *** ## Key Generation ### generateKey() Generate a random 256-bit key. ```typescript theme={null} import { generateKey } from 'rehydra'; const key = generateKey(); // Uint8Array (32 bytes) ``` ### generateSalt() Generate a random salt for key derivation. ```typescript theme={null} import { generateSalt } from 'rehydra'; const salt = generateSalt(); // Uint8Array (16 bytes) ``` ### deriveKey() Derive a key from a password using PBKDF2. ```typescript theme={null} import { deriveKey, generateSalt } from 'rehydra'; const password = 'user-password'; const salt = generateSalt(); const key = await deriveKey(password, salt); // Store salt with encrypted data for later derivation ``` *** ## Utility Functions ### validateKey() Check if a key is valid. ```typescript theme={null} import { validateKey } from 'rehydra'; if (validateKey(key)) { console.log('Key is valid'); } else { throw new Error('Invalid key'); } ``` ### secureCompare() Constant-time comparison to prevent timing attacks. ```typescript theme={null} import { secureCompare } from 'rehydra'; if (secureCompare(key1, key2)) { console.log('Keys match'); } ``` ### uint8ArrayToBase64() Convert Uint8Array to Base64. ```typescript theme={null} import { uint8ArrayToBase64 } from 'rehydra'; const base64 = uint8ArrayToBase64(key); // Store this string securely ``` ### base64ToUint8Array() Convert Base64 to Uint8Array. ```typescript theme={null} import { base64ToUint8Array } from 'rehydra'; const key = base64ToUint8Array(base64); ``` *** ## EncryptedPIIMap Structure ```typescript theme={null} interface EncryptedPIIMap { ciphertext: string; // Base64-encoded encrypted data iv: string; // Base64-encoded initialization vector authTag: string; // Base64-encoded authentication tag } ``` *** ## Complete Example ```typescript theme={null} import { generateKey, encryptPIIMap, decryptPIIMap, uint8ArrayToBase64, base64ToUint8Array } from 'rehydra'; // Generate and store key const key = generateKey(); const keyBase64 = uint8ArrayToBase64(key); // Store keyBase64 securely (env var, keychain, etc.) // Later: encrypt const piiMap = new Map([['EMAIL:1', 'secret@example.com']]); const encrypted = await encryptPIIMap(piiMap, key); // Later: decrypt const restoredKey = base64ToUint8Array(keyBase64); const decrypted = await decryptPIIMap(encrypted, restoredKey); ``` *** ## Security Notes * Never commit keys to version control * Use environment variables or secrets managers * Consider platform keystores (iOS Keychain, Android Keystore) Implement key rotation for long-running applications: ```typescript theme={null} // Re-encrypt with new key const piiMap = await decryptPIIMap(encrypted, oldKey); const newEncrypted = await encryptPIIMap(piiMap, newKey); ``` In browsers: * Keys in memory can be accessed by malicious scripts * Consider using SubtleCrypto key wrapping * Clear keys when no longer needed ## Related * [Encryption Concept](/concepts/encryption) - How encryption works * [createAnonymizer](/api-reference/create-anonymizer) - Key provider configuration # API Reference Source: https://docs.rehydra.ai/api-reference/introduction Complete API documentation for Rehydra This section provides detailed documentation for all Rehydra exports. ## Core Functions Create a reusable anonymizer instance with configuration Anonymize text and get the result with encrypted PII map Restore original values from anonymized text ## Storage SQLite, IndexedDB, and in-memory storage Session-based PII map persistence ## Streaming & Proxy Stream-based anonymization for chunked text (Node.js/Bun) Proxy middleware for OpenAI and Anthropic (Node.js/Bun) ## Utilities Encryption, key management, and utilities Download, cache, and manage NER models ## Quick Reference ### Main Entry Points ```typescript theme={null} import { // Factory function createAnonymizer, // Convenience functions anonymize, anonymizeRegexOnly, anonymizeWithNER, // Rehydration rehydrate, decryptPIIMap, // Key providers InMemoryKeyProvider, ConfigKeyProvider, // Storage providers InMemoryPIIStorageProvider, SQLitePIIStorageProvider, // Node.js only IndexedDBPIIStorageProvider, // Browser only // Types PIIType, type AnonymizationResult, type AnonymizationPolicy, } from 'rehydra'; // Streaming (Node.js/Bun only) import { createAnonymizerStream } from 'rehydra/streaming'; // LLM Proxy (Node.js/Bun only) import { wrapLLMClient, createRehydraFetch, createRehydraProxy } from 'rehydra/proxy'; ``` ### Result Structure ```typescript theme={null} interface AnonymizationResult { // Text with PII replaced anonymizedText: string; // Detected entities (without originals) entities: Array<{ type: PIIType; id: number; start: number; end: number; confidence: number; source: 'REGEX' | 'NER' | 'HYBRID'; semantic?: { gender?: string; scope?: string; title?: string }; }>; // Encrypted PII map (undefined in 'anonymize' mode) piiMap?: { ciphertext: string; iv: string; authTag: string; }; // Statistics stats: { countsByType: Record; totalEntities: number; processingTimeMs: number; modelVersion: string; leakScanPassed?: boolean; }; } ``` ### Configuration ```typescript theme={null} interface AnonymizerConfig { // Anonymization mode mode?: 'pseudonymize' | 'anonymize'; // default: 'pseudonymize' // 'pseudonymize': reversible — returns encrypted piiMap for rehydration // 'anonymize': irreversible — no piiMap returned, PII is permanently removed // NER model configuration ner?: { mode: 'disabled' | 'quantized' | 'standard' | 'custom'; modelPath?: string; vocabPath?: string; autoDownload?: boolean; thresholds?: Partial>; onStatus?: (status: string) => void; onDownloadProgress?: (progress: DownloadProgress) => void; // Case-insensitive fallback for lowercase names caseFallback?: boolean; // default: false caseFallbackPenalty?: number; // default: 0.85 // Inference backend backend?: 'local' | 'inference-server'; // default: 'local' inferenceServerUrl?: string; // required if backend='inference-server' inferenceServerTimeout?: number; // default: 30000 }; // Semantic enrichment semantic?: { enabled: boolean; autoDownload?: boolean; onStatus?: (status: string) => void; onDownloadProgress?: (progress: DownloadProgress) => void; }; // Encryption key provider keyProvider?: KeyProvider; // Storage for sessions piiStorageProvider?: PIIStorageProvider; // Default anonymization policy defaultPolicy?: AnonymizationPolicy; // Secrets/credentials detection secrets?: { enabled: boolean; envFiles?: string[]; secretKeyPatterns?: RegExp[]; minValueLength?: number; // default: 4 }; // Validation warning callback (silent if omitted) onValidationWarning?: (warnings: Array<{ code: string; message: string }>) => void; } ``` ### Policy Options ```typescript theme={null} interface AnonymizationPolicy { // Types to detect enabledTypes: Set; regexEnabledTypes: Set; nerEnabledTypes: Set; // Confidence thresholds confidenceThresholds: Map; // Type priority for overlaps typePriority: PIIType[]; // Custom ID patterns customIdPatterns: CustomIdPattern[]; // Allowlist (never anonymize these terms) allowlistTerms: Set; // Denylist (always anonymize matches) denylistPatterns: RegExp[]; // ID reuse for repeated values reuseIdsForRepeatedPII: boolean; // Validation enableLeakScan: boolean; // Semantic attributes enableSemanticMasking: boolean; } ``` # Model Management Source: https://docs.rehydra.ai/api-reference/model-management Download, cache, and manage NER models Rehydra provides utilities for managing NER model downloads and cache. ## Model Information ### Available Models | Mode | Model Name | Size | Description | | ------------- | ---------- | -------- | ------------------------- | | `'quantized'` | quantized | \~280 MB | Smaller, \~95% accuracy | | `'standard'` | standard | \~1.1 GB | Full model, best accuracy | ### Model Files Each model includes: * `model.onnx` - The ONNX model file * `vocab.txt` - WordPiece vocabulary * `label_map.json` - Entity label mapping *** ## Check Model Status ### isModelDownloaded() Check if a model is cached locally. ```typescript theme={null} import { isModelDownloaded } from 'rehydra'; const hasQuantized = await isModelDownloaded('quantized'); const hasStandard = await isModelDownloaded('standard'); console.log(`Quantized: ${hasQuantized}, Standard: ${hasStandard}`); ``` ### listDownloadedModels() List all cached models. ```typescript theme={null} import { listDownloadedModels } from 'rehydra'; const models = await listDownloadedModels(); // ['quantized'] or ['quantized', 'standard'] or [] ``` *** ## Download Models ### downloadModel() Manually download a model. ```typescript theme={null} import { downloadModel } from 'rehydra'; await downloadModel('quantized', (progress) => { console.log(`${progress.file}: ${progress.percent}%`); }); ``` ### ensureModel() Download if not present, return paths. ```typescript theme={null} import { ensureModel } from 'rehydra'; const { modelPath, vocabPath, labelMapPath } = await ensureModel('quantized', { autoDownload: true, onProgress: (p) => console.log(`${p.file}: ${p.percent}%`), onStatus: (s) => console.log(s), }); ``` *** ## Clear Cache ### clearModelCache() Remove cached models. ```typescript theme={null} import { clearModelCache } from 'rehydra'; // Clear specific model await clearModelCache('quantized'); // Clear all models await clearModelCache(); ``` *** ## Cache Locations ### getModelCacheDir() Get the cache directory path. ```typescript theme={null} import { getModelCacheDir } from 'rehydra'; const cacheDir = getModelCacheDir(); ``` ### Node.js Locations | Platform | Location | | -------- | ---------------------------------- | | macOS | `~/Library/Caches/rehydra/models/` | | Linux | `~/.cache/rehydra/models/` | | Windows | `%LOCALAPPDATA%/rehydra/models/` | ### Browser Storage * **OPFS (Origin Private File System)** for model files * Data persists across sessions *** ## Semantic Data Management Similar functions exist for semantic enrichment data: ```typescript theme={null} import { isSemanticDataDownloaded, downloadSemanticData, clearSemanticDataCache, getSemanticDataCacheDir, getSemanticDataInfo, } from 'rehydra'; // Check status const hasData = await isSemanticDataDownloaded(); // Download with progress await downloadSemanticData((progress) => { console.log(`${progress.file}: ${progress.percent}%`); }); // Get info const info = getSemanticDataInfo(); // { files: [...], totalSize: 12000000 } // Clear cache await clearSemanticDataCache(); ``` *** ## Download Progress Callback ```typescript theme={null} interface DownloadProgress { file: string; // Current file name percent: number; // 0-100 loaded: number; // Bytes loaded total: number; // Total bytes } ``` Example usage: ```typescript theme={null} const anonymizer = createAnonymizer({ ner: { mode: 'quantized', onDownloadProgress: (progress) => { const bar = '█'.repeat(progress.percent / 5) + '░'.repeat(20 - progress.percent / 5); console.log(`${progress.file}: [${bar}] ${progress.percent}%`); } } }); ``` *** ## Pre-Warming Download models before user interaction: ```typescript theme={null} // On app start async function preWarmModels() { const hasModel = await isModelDownloaded('quantized'); if (!hasModel) { console.log('Downloading NER model...'); await downloadModel('quantized', console.log); } const hasSemanticData = await isSemanticDataDownloaded(); if (!hasSemanticData) { console.log('Downloading semantic data...'); await downloadSemanticData(console.log); } console.log('Models ready!'); } ``` *** ## Model Registry Access model metadata: ```typescript theme={null} import { MODEL_REGISTRY } from 'rehydra'; console.log(MODEL_REGISTRY); // { // quantized: { // name: 'quantized', // files: ['model.onnx', 'vocab.txt', 'label_map.json'], // baseUrl: 'https://...', // ... // }, // standard: { ... } // } ``` *** ## Offline Usage After initial download, models work offline: ```typescript theme={null} // First run: downloads (~30 seconds on fast connection) await anonymizer.initialize(); // Subsequent runs: instant (cached) await anonymizer.initialize(); // ~100ms ``` *** ## Custom Models Use your own ONNX model: ```typescript theme={null} const anonymizer = createAnonymizer({ ner: { mode: 'custom', modelPath: './my-model.onnx', vocabPath: './my-vocab.txt', } }); ``` ### Model Requirements * **Format**: ONNX * **Input**: Token IDs, attention mask * **Output**: Logits for BIO-tagged entities * **Vocab**: WordPiece format See [Building Custom Models](https://github.com/rehydra-ai/rehydra-sdk#building-custom-models) for details. ## Related * [NER Detection Guide](/guides/ner-detection) - NER configuration * [createAnonymizer](/api-reference/create-anonymizer) - NER config options # LLM Proxy Source: https://docs.rehydra.ai/api-reference/proxy Proxy middleware API reference for OpenAI and Anthropic Transparent PII anonymization and rehydration for LLM API calls. Available in Node.js and Bun only. ## Import ```typescript theme={null} import { wrapLLMClient, createRehydraFetch, createRehydraProxy, createRehydraProxyServer, SSEParser, OpenAIProvider, AnthropicProvider, detectProvider, } from 'rehydra/proxy'; ``` *** ## wrapLLMClient() Wraps an OpenAI or Anthropic SDK client for automatic anonymization/rehydration. ### Signature ```typescript theme={null} function wrapLLMClient( client: T, config: Omit ): T ``` ### Parameters An OpenAI or Anthropic SDK client instance Configuration (see [RehydraFetchConfig](#rehydrafetchconfig)) ### Returns The same client type `T`, with fetch intercepted for anonymization. ### Example ```typescript theme={null} import OpenAI from 'openai'; import { wrapLLMClient } from 'rehydra/proxy'; import { InMemoryKeyProvider, SQLitePIIStorageProvider } from 'rehydra'; const storage = new SQLitePIIStorageProvider('./pii.db'); await storage.initialize(); const client = wrapLLMClient(new OpenAI(), { anonymizer: { ner: { mode: 'quantized' } }, keyProvider: new InMemoryKeyProvider(), piiStorageProvider: storage, getSessionId: () => 'session-1', }); ``` *** ## createRehydraFetch() Creates a drop-in `fetch` replacement that anonymizes requests and rehydrates responses. ### Signature ```typescript theme={null} function createRehydraFetch(config: RehydraFetchConfig): typeof globalThis.fetch ``` ### Parameters Configuration (see [RehydraFetchConfig](#rehydrafetchconfig)) ### Returns A `fetch`-compatible function. ### Example ```typescript theme={null} import { createRehydraFetch } from 'rehydra/proxy'; import { InMemoryKeyProvider, SQLitePIIStorageProvider } from 'rehydra'; const storage = new SQLitePIIStorageProvider('./pii.db'); await storage.initialize(); const fetch = createRehydraFetch({ anonymizer: { ner: { mode: 'quantized' } }, keyProvider: new InMemoryKeyProvider(), piiStorageProvider: storage, provider: 'openai', // or 'anthropic', or auto-detected getSessionId: () => 'session-1', }); const response = await fetch('https://api.openai.com/v1/chat/completions', { method: 'POST', body: JSON.stringify({ model: 'gpt-4', messages: [...] }), }); ``` *** ## createRehydraProxy() Creates an HTTP proxy middleware using the Web Request/Response API. ### Signature ```typescript theme={null} function createRehydraProxy( config: RehydraProxyConfig ): (request: Request) => Promise ``` ### Parameters Configuration (see [RehydraProxyConfig](#rehydraproxyconfig)) ### Returns A function `(request: Request) => Promise` compatible with Bun.serve, Hono, and other frameworks using Web APIs. ### Example ```typescript theme={null} import { createRehydraProxy } from 'rehydra/proxy'; import { InMemoryKeyProvider, SQLitePIIStorageProvider } from 'rehydra'; const storage = new SQLitePIIStorageProvider('./pii.db'); await storage.initialize(); const proxy = createRehydraProxy({ upstream: 'https://api.openai.com', anonymizer: { ner: { mode: 'quantized' } }, keyProvider: new InMemoryKeyProvider(), piiStorageProvider: storage, getSessionId: () => 'session-1', }); Bun.serve({ port: 8080, fetch: proxy }); ``` *** ## createRehydraProxyServer() Creates and starts a standalone HTTP proxy server. ### Signature ```typescript theme={null} function createRehydraProxyServer( config: RehydraProxyServerConfig ): Promise ``` ### Parameters Extends [RehydraProxyConfig](#rehydraproxyconfig) with: Port to listen on Host to bind to ### Returns ```typescript theme={null} interface RehydraProxyServer { server: http.Server; port: number; host: string; close(): Promise; } ``` ### Example ```typescript theme={null} import { createRehydraProxyServer } from 'rehydra/proxy'; import { InMemoryKeyProvider, SQLitePIIStorageProvider } from 'rehydra'; const storage = new SQLitePIIStorageProvider('./pii.db'); await storage.initialize(); const server = await createRehydraProxyServer({ upstream: 'https://api.openai.com', port: 8080, anonymizer: { ner: { mode: 'quantized' } }, keyProvider: new InMemoryKeyProvider(), piiStorageProvider: storage, getSessionId: () => 'session-1', }); // Point clients at http://localhost:8080/v1 await server.close(); ``` *** ## Configuration Types ### RehydraFetchConfig ```typescript theme={null} interface RehydraFetchConfig { // Encryption key provider (required) keyProvider: KeyProvider; // Storage for session persistence (required) piiStorageProvider: PIIStorageProvider; // Anonymizer configuration (includes tagFormat) anonymizer?: AnonymizerConfig; // Policy override policy?: Partial; // Provider hint (auto-detected if omitted) provider?: 'openai' | 'anthropic' | 'auto'; // Derive session ID from request getSessionId?: (request: Request) => string | Promise; // Handle streaming responses (default: true) handleStreaming?: boolean; // Locale hint locale?: string; // Tool execution callback (non-streaming only) // Arguments are rehydrated before calling; return value is anonymized onToolCall?: ( name: string, args: Record, toolCallId: string, ) => unknown | Promise; // Maximum tool execution rounds (default: 10) maxToolRounds?: number; // PII system instruction: string to override, false to disable, undefined for default systemInstruction?: string | false; } ``` ### RehydraProxyConfig Extends `RehydraFetchConfig` with: ```typescript theme={null} interface RehydraProxyConfig extends RehydraFetchConfig { // Upstream LLM API URL (e.g., 'https://api.openai.com') upstream: string; // Headers to forward to upstream // Default: ['authorization', 'content-type', 'x-api-key', // 'anthropic-version', 'openai-organization', 'openai-project'] forwardHeaders?: string[]; // Strip this prefix from the request path before forwarding stripPrefix?: string; // API key to inject into upstream requests // Adds Authorization header (OpenAI) or x-api-key header (Anthropic) apiKey?: string; } ``` *** ## Providers ### detectProvider() Auto-detect the LLM provider from request URL and headers. ```typescript theme={null} function detectProvider( url: string, headers: Headers, hint?: string ): LLMContentProvider ``` ### LLMContentProvider Interface ```typescript theme={null} interface LLMContentProvider { readonly name: string; // Request detection matchesRequest(url: string, headers: Headers): boolean; isStreamingRequest(body: unknown): boolean; // Request handling extractRequestText(body: unknown): string[]; rebuildRequestBody(body: unknown, anonymizedTexts: string[]): unknown; // Non-streaming response handling extractResponseText(body: unknown): string[]; rebuildResponseBody(body: unknown, rehydratedTexts: string[]): unknown; // Streaming content handling extractSSEDelta(data: unknown): string | null; rebuildSSEDelta(data: unknown, rehydratedText: string): unknown; // Tool/function call handling (non-streaming) extractResponseToolCalls?(body: unknown): string[]; rebuildResponseToolCalls?(body: unknown, rehydratedArgs: string[]): unknown; // Tool/function call handling (streaming) extractSSEToolCallDeltas?(data: unknown): ToolCallDelta[] | null; rebuildSSEToolCallDeltas?(data: unknown, rehydratedArgs: Map): unknown; extractSSEToolCallStop?(data: unknown): number | null; } interface ToolCallDelta { index: number; arguments: string; } interface SSEEvent { event: string; // defaults to 'message' data: string; } ``` ### Built-in Providers | Provider | Matches | | ------------------- | ------------------------------------------------------------------ | | `OpenAIProvider` | `api.openai.com` URL or `Bearer sk-` auth header | | `AnthropicProvider` | `api.anthropic.com` URL or `x-api-key`/`anthropic-version` headers | *** ## SSEParser Parser for Server-Sent Events streams. ```typescript theme={null} const parser = new SSEParser(); // Parse SSE text const events = parser.parse(chunk); // [{ event: 'message', data: '...' }, ...] // Check for stream end import { isSSEDone } from 'rehydra/proxy'; if (isSSEDone(event.data)) { /* stream complete */ } // Serialize back to SSE format import { serializeSSEEvent } from 'rehydra/proxy'; const text = serializeSSEEvent({ event: 'message', data: '...' }); ``` ## Related * [LLM Proxy Guide](/guides/llm-proxy) - Usage guide and examples * [Streaming Guide](/guides/streaming) - Stream-level anonymization # Rehydrate Functions Source: https://docs.rehydra.ai/api-reference/rehydrate Restore original PII values from anonymized text Rehydration is the process of replacing placeholder tags with original values. ## rehydrate() Replaces PII placeholders with original values. ### Signature ```typescript theme={null} function rehydrate( text: string, piiMap: Map, strict?: boolean, tagFormat?: TagFormat ): string ``` ### Parameters Text containing PII placeholder tags Decrypted PII map from `decryptPIIMap()` If `true`, use strict matching (exact tag format only). If `false`, use fuzzy matching that handles mangled tags from translation. Tag format configuration. Must match the format used during anonymization. Defaults to XML-style (``). ### Returns Text with placeholders replaced by original values. ### Example ```typescript theme={null} import { rehydrate, decryptPIIMap } from 'rehydra'; // After anonymization and translation const translated = 'Hallo von !'; // Decrypt the PII map const key = await keyProvider.getKey(); const piiMap = await decryptPIIMap(result.piiMap, key); // Map { 'PERSON:1' => 'John Smith', 'ORG:1' => 'Acme Corp' } // Rehydrate const original = rehydrate(translated, piiMap); // "Hallo John Smith von Acme Corp!" ``` *** ## decryptPIIMap() Decrypts the encrypted PII map from an anonymization result. ### Signature ```typescript theme={null} function decryptPIIMap( encryptedMap: EncryptedPIIMap, key: Uint8Array ): Promise> ``` ### Parameters The encrypted PII map from `result.piiMap` ```typescript theme={null} interface EncryptedPIIMap { ciphertext: string; // Base64 iv: string; // Base64 authTag: string; // Base64 } ``` The encryption key (same key used during anonymization) ### Returns A `Map` where: * Key: `"TYPE:id"` (e.g., `"EMAIL:1"`, `"PERSON:2"`) * Value: Original PII value ### Example ```typescript theme={null} import { decryptPIIMap, InMemoryKeyProvider } from 'rehydra'; const keyProvider = new InMemoryKeyProvider(); // During anonymization const anonymizer = createAnonymizer({ keyProvider }); const result = await anonymizer.anonymize('Email: john@example.com'); // Later, to rehydrate const key = await keyProvider.getKey(); const piiMap = await decryptPIIMap(result.piiMap, key); console.log(piiMap.get('EMAIL:1')); // "john@example.com" ``` *** ## Complete Workflow ```typescript theme={null} import { createAnonymizer, decryptPIIMap, rehydrate, InMemoryKeyProvider } from 'rehydra'; // 1. Setup const keyProvider = new InMemoryKeyProvider(); const anonymizer = createAnonymizer({ ner: { mode: 'quantized' }, keyProvider }); await anonymizer.initialize(); // 2. Anonymize const result = await anonymizer.anonymize( 'Contact John Smith at john@acme.com' ); console.log(result.anonymizedText); // "Contact at " // 3. Process (e.g., translate) const translated = await translate(result.anonymizedText); // "Kontaktieren Sie unter " // 4. Decrypt PII map const key = await keyProvider.getKey(); const piiMap = await decryptPIIMap(result.piiMap, key); // 5. Rehydrate const final = rehydrate(translated, piiMap); console.log(final); // "Kontaktieren Sie John Smith unter john@acme.com" // 6. Cleanup await anonymizer.dispose(); ``` *** ## Session-Based Rehydration When using sessions, rehydration is simplified: ```typescript theme={null} const session = anonymizer.session('chat-123'); // Anonymize (auto-saves PII map) const result = await session.anonymize('Hello John Smith!'); // Rehydrate (auto-loads and decrypts) const original = await session.rehydrate(result.anonymizedText); ``` See [Sessions](/api-reference/sessions) for details. *** ## Placeholder Format By default, rehydrate expects placeholders in XML-style format: ```xml theme={null} ``` When using a custom tag format, pass the same `tagFormat` used during anonymization: ```typescript theme={null} const tagFormat = { open: '[[', close: ']]' }; const text = '[[PII type="PERSON" id="1"]] from [[PII type="ORG" id="2"]]'; const result = rehydrate(text, piiMap, false, tagFormat); ``` The function matches based on `type` and `id` attributes. ### Handling Malformed Placeholders ```typescript theme={null} const text = 'Hello and '; const result = rehydrate(text, piiMap); // "Hello John Smith and " // Unmatched tags are preserved ``` *** ## Error Handling ### Missing Key ```typescript theme={null} try { const piiMap = await decryptPIIMap(result.piiMap, wrongKey); } catch (error) { // "Decryption failed: invalid key or corrupted data" } ``` ### Missing PII Entry ```typescript theme={null} const piiMap = new Map([['EMAIL:1', 'test@example.com']]); const text = ''; // Not in map const result = rehydrate(text, piiMap); // "" // Unchanged (no match) ``` ## Related * [Encryption](/concepts/encryption) - How PII maps are encrypted * [Sessions](/api-reference/sessions) - Simplified rehydration with sessions * [Crypto](/api-reference/crypto) - Key management utilities # Sessions Source: https://docs.rehydra.ai/api-reference/sessions Session-based API for multi-turn conversations Sessions provide a simplified API for anonymizing and rehydrating text across multiple turns, with automatic PII map persistence. ## Creating a Session ```typescript theme={null} import { createAnonymizer, InMemoryKeyProvider, SQLitePIIStorageProvider } from 'rehydra'; // Setup storage const storage = new SQLitePIIStorageProvider('./pii.db'); await storage.initialize(); // Create anonymizer with storage const anonymizer = createAnonymizer({ ner: { mode: 'quantized' }, keyProvider: new InMemoryKeyProvider(), piiStorageProvider: storage, }); await anonymizer.initialize(); // Create a session const session = anonymizer.session('conversation-123'); ``` Both `keyProvider` and `piiStorageProvider` are required for sessions. Without them, `session()` will throw an error. *** ## AnonymizerSession Interface ```typescript theme={null} interface AnonymizerSession { /** * The session ID */ readonly sessionId: string; /** * Anonymize text and save PII map to storage */ anonymize( text: string, locale?: string, policy?: Partial ): Promise; /** * Load PII map from storage and restore original values */ rehydrate(text: string): Promise; /** * Load the stored PII map */ load(): Promise; /** * Delete this session from storage */ delete(): Promise; /** * Check if this session exists in storage */ exists(): Promise; /** * Recursively anonymize all string leaves in a JSON-serializable value */ anonymizeJson( value: T, locale?: string, policy?: Partial ): Promise; /** * Recursively rehydrate all string leaves in a JSON-serializable value */ rehydrateJson(value: T): Promise; } ``` *** ## session.anonymize() Anonymizes text and automatically saves the PII map. ```typescript theme={null} const session = anonymizer.session('chat-123'); const result = await session.anonymize('Hello John Smith!'); console.log(result.anonymizedText); // "Hello !" // PII map is automatically saved to storage ``` ### Multi-Message Consistency Entity IDs are consistent across multiple calls: ```typescript theme={null} // Message 1 const msg1 = await session.anonymize('Email me at john@example.com'); // "Email me at " // Message 2 - same email gets same ID const msg2 = await session.anonymize('Or use john@example.com directly'); // "Or use directly" // ↑ Same ID ``` *** ## session.rehydrate() Loads the PII map from storage and restores original values. ```typescript theme={null} const session = anonymizer.session('chat-123'); // After anonymization and processing const translated = 'Hallo !'; const original = await session.rehydrate(translated); console.log(original); // "Hallo John Smith!" ``` ### Cross-Request Rehydration ```typescript theme={null} // Request 1: Anonymize const session1 = anonymizer.session('chat-123'); await session1.anonymize('Hello John Smith!'); // Request 2 (later, even after server restart): Rehydrate const session2 = anonymizer.session('chat-123'); // Same ID const original = await session2.rehydrate(translatedText); ``` *** ## session.load() Manually load the stored PII map. ```typescript theme={null} const session = anonymizer.session('chat-123'); const storedMap = await session.load(); if (storedMap) { console.log('Entity count:', storedMap.metadata.entityCount); console.log('Types:', storedMap.metadata.types); console.log('Created:', storedMap.metadata.createdAt); } ``` *** ## session.exists() Check if a session has stored data. ```typescript theme={null} const session = anonymizer.session('chat-123'); if (await session.exists()) { console.log('Session has stored data'); } else { console.log('New session'); } ``` *** ## session.delete() Remove a session from storage. ```typescript theme={null} const session = anonymizer.session('chat-123'); const wasDeleted = await session.delete(); console.log(wasDeleted ? 'Deleted' : 'Not found'); ``` *** ## session.anonymizeJson() Recursively walks a JSON-serializable value and anonymizes every string leaf. Non-string values (numbers, booleans, nulls) pass through unchanged. The PII map is saved to storage after processing. ```typescript theme={null} const session = anonymizer.session('chat-123'); const toolArgs = { to: 'john@acme.com', subject: 'Meeting with Alice', body: 'Hi Alice, call me at 555-0123.', }; const anonymized = await session.anonymizeJson(toolArgs); // { // to: '', // subject: 'Meeting with ', // body: 'Hi , call me at .', // } ``` Strings are processed **sequentially** (not in parallel) to ensure deterministic PII ID assignment. *** ## session.rehydrateJson() Recursively walks a JSON-serializable value and rehydrates every string leaf that contains PII placeholders. ```typescript theme={null} const session = anonymizer.session('chat-123'); const anonymizedResult = { status: 'sent', recipient: '', message: 'Email sent to ', }; const rehydrated = await session.rehydrateJson(anonymizedResult); // { // status: 'sent', // recipient: 'john@acme.com', // message: 'Email sent to Alice', // } ``` *** ## Complete Example ```typescript theme={null} import { createAnonymizer, InMemoryKeyProvider, SQLitePIIStorageProvider } from 'rehydra'; async function main() { // Setup const storage = new SQLitePIIStorageProvider('./pii.db'); await storage.initialize(); const anonymizer = createAnonymizer({ ner: { mode: 'quantized' }, keyProvider: new InMemoryKeyProvider(), piiStorageProvider: storage, }); await anonymizer.initialize(); // Conversation session const session = anonymizer.session('user-alice-conversation'); // Turn 1 const turn1 = await session.anonymize('I am Alice from Berlin'); console.log(turn1.anonymizedText); // "I am from " // Simulate translation const translated1 = await translate(turn1.anonymizedText); // Turn 2 const turn2 = await session.anonymize('Alice works at Acme Corp'); console.log(turn2.anonymizedText); // "I am works at " // ↑ Same ID (Alice) // Rehydrate both turns const original1 = await session.rehydrate(translated1); const original2 = await session.rehydrate(translated2); // Cleanup old sessions const weekAgo = new Date(Date.now() - 7 * 24 * 60 * 60 * 1000); await storage.cleanup(weekAgo); // Shutdown await anonymizer.dispose(); await storage.close(); } ``` *** ## Multiple Conversations ```typescript theme={null} // Each user gets their own session function handleUserMessage(userId: string, message: string) { const session = anonymizer.session(`user-${userId}`); return session.anonymize(message); } // Sessions are independent await handleUserMessage('alice', 'I am Alice'); // Session: user-alice await handleUserMessage('bob', 'I am Bob'); // Session: user-bob ``` *** ## Session vs Direct Anonymize | Feature | `anonymizer.anonymize()` | `session.anonymize()` | | --------------------------- | ------------------------ | --------------------- | | Auto-save PII map | ❌ | ✅ | | Cross-request rehydration | ❌ (manual) | ✅ | | ID consistency across calls | ❌ | ✅ | | Requires storage provider | ❌ | ✅ | ## Related * [Storage Providers](/api-reference/storage-providers) - Available storage options * [Sessions & Storage Guide](/guides/sessions-storage) - Detailed usage guide * [createAnonymizer](/api-reference/create-anonymizer) - Configuration options # Storage Providers Source: https://docs.rehydra.ai/api-reference/storage-providers Persist encrypted PII maps across requests Storage providers enable persistent storage of encrypted PII maps for multi-turn conversations or cross-request rehydration. ## Available Providers | Provider | Environment | Persistence | | ----------------------------- | ------------ | ---------------------- | | `InMemoryPIIStorageProvider` | All | None (lost on restart) | | `SQLitePIIStorageProvider` | Node.js, Bun | File-based | | `IndexedDBPIIStorageProvider` | Browser | Browser storage | *** ## InMemoryPIIStorageProvider Simple in-memory storage for development and testing. ```typescript theme={null} import { InMemoryPIIStorageProvider } from 'rehydra'; const storage = new InMemoryPIIStorageProvider(); // No initialization needed ``` Data is lost when the process ends. Use only for development/testing. *** ## SQLitePIIStorageProvider Persistent file-based storage for Node.js and Bun. ### Import ```typescript theme={null} import { SQLitePIIStorageProvider } from 'rehydra'; // Or explicitly: import { SQLitePIIStorageProvider } from 'rehydra/storage/sqlite'; ``` ### Usage ```typescript theme={null} // File-based database const storage = new SQLitePIIStorageProvider('./data/pii-maps.db'); await storage.initialize(); // In-memory database (for testing) const testStorage = new SQLitePIIStorageProvider(':memory:'); await testStorage.initialize(); ``` ### Dependencies | Runtime | Dependency | | ------- | ----------------------------------------- | | Bun | Built-in `bun:sqlite` (no install needed) | | Node.js | `better-sqlite3` | ```bash theme={null} # Node.js only npm install better-sqlite3 ``` `SQLitePIIStorageProvider` is **not available in browser builds**. The browser entry point excludes it to avoid bundling Node.js dependencies. *** ## IndexedDBPIIStorageProvider Browser-native storage using IndexedDB. ```typescript theme={null} import { IndexedDBPIIStorageProvider } from 'rehydra'; // Default database name const storage = new IndexedDBPIIStorageProvider(); // Custom database name const storage = new IndexedDBPIIStorageProvider('my-app-pii'); ``` ### Features * Persists across page reloads * Persists across browser sessions * Cleared with "Clear site data" * Quota managed by browser *** ## PIIStorageProvider Interface All providers implement this interface: ```typescript theme={null} interface PIIStorageProvider { /** * Initialize the storage (create tables, etc.) */ initialize(): Promise; /** * Save a PII map */ save(sessionId: string, data: StoredPIIMap): Promise; /** * Load a PII map */ load(sessionId: string): Promise; /** * Delete a PII map */ delete(sessionId: string): Promise; /** * Check if a session exists */ exists(sessionId: string): Promise; /** * List all session IDs */ list(options?: ListOptions): Promise; /** * Delete entries older than a date */ cleanup(olderThan: Date): Promise; /** * Close the storage connection */ close(): Promise; } ``` *** ## StoredPIIMap Structure ```typescript theme={null} interface StoredPIIMap { // Encrypted PII data encrypted: { ciphertext: string; iv: string; authTag: string; }; // Metadata metadata: { createdAt: Date; updatedAt: Date; entityCount: number; types: PIIType[]; }; } ``` *** ## Integration with Anonymizer ```typescript theme={null} import { createAnonymizer, InMemoryKeyProvider, SQLitePIIStorageProvider } from 'rehydra'; const storage = new SQLitePIIStorageProvider('./pii.db'); await storage.initialize(); const anonymizer = createAnonymizer({ ner: { mode: 'quantized' }, keyProvider: new InMemoryKeyProvider(), piiStorageProvider: storage, // Enable session support }); await anonymizer.initialize(); // Now you can use sessions const session = anonymizer.session('conversation-id'); ``` *** ## Direct Storage Operations You can use storage providers directly: ```typescript theme={null} const storage = new SQLitePIIStorageProvider('./pii.db'); await storage.initialize(); // List all sessions const sessionIds = await storage.list(); // List with pagination const page = await storage.list({ limit: 10, offset: 0 }); // Check existence if (await storage.exists('session-123')) { const data = await storage.load('session-123'); } // Delete old data const deleted = await storage.cleanup(new Date(Date.now() - 7 * 24 * 60 * 60 * 1000)); console.log(`Deleted ${deleted} old entries`); // Close connection await storage.close(); ``` *** ## Custom Storage Provider Implement the interface for custom storage: ```typescript theme={null} import { PIIStorageProvider, StoredPIIMap } from 'rehydra'; class RedisStorageProvider implements PIIStorageProvider { private client: RedisClient; async initialize(): Promise { this.client = await createRedisClient(); } async save(sessionId: string, data: StoredPIIMap): Promise { await this.client.set(`pii:${sessionId}`, JSON.stringify(data)); } async load(sessionId: string): Promise { const raw = await this.client.get(`pii:${sessionId}`); return raw ? JSON.parse(raw) : null; } async delete(sessionId: string): Promise { return (await this.client.del(`pii:${sessionId}`)) > 0; } async exists(sessionId: string): Promise { return (await this.client.exists(`pii:${sessionId}`)) > 0; } async list(): Promise { const keys = await this.client.keys('pii:*'); return keys.map(k => k.replace('pii:', '')); } async cleanup(olderThan: Date): Promise { // Implement based on your needs return 0; } async close(): Promise { await this.client.quit(); } } ``` ## Related * [Sessions](/api-reference/sessions) - Session-based API * [Sessions & Storage Guide](/guides/sessions-storage) - Usage guide * [Browser Usage](/guides/browser-usage) - Browser-specific storage # Streaming Source: https://docs.rehydra.ai/api-reference/streaming Streaming anonymization API reference Stream-based anonymization for processing text chunk by chunk. Available in Node.js and Bun only. ## Import ```typescript theme={null} import { createAnonymizerStream, AnonymizerStream, SentenceBuffer } from 'rehydra/streaming'; ``` *** ## createAnonymizerStream() Factory function that creates and initializes a streaming anonymizer. ### Signature ```typescript theme={null} function createAnonymizerStream(config?: StreamConfig): Promise ``` ### Parameters Anonymizer configuration (same options as `createAnonymizer`) Policy override for the stream Locale hint (e.g., `'de-DE'`) Buffer configuration for sentence detection Characters of overlap between buffer flushes for NER context Force flush when buffer reaches this size Minimum buffer size before allowing flush Enable low-latency mode (disables NER, reduces buffer sizes) Custom regex for sentence boundary detection Session ID for PII map persistence Encryption key provider (required for session persistence) Storage provider (required for session persistence) Interval for periodic PII map saves (milliseconds) Node.js Transform stream options Callback after each chunk is anonymized Callback when the stream ends ### Returns `Promise` — a Node.js Transform stream. ### Example ```typescript theme={null} import { createAnonymizerStream } from 'rehydra/streaming'; const stream = await createAnonymizerStream({ anonymizer: { ner: { mode: 'quantized' } }, onChunk: (e) => console.log(`Processed: ${e.totalEntities} entities`), onFinish: (e) => console.log(`Done: ${e.totalEntities} total`), }); process.stdin.pipe(stream).pipe(process.stdout); ``` *** ## AnonymizerStream A Node.js `Transform` stream that anonymizes text passing through it. ### Methods | Method | Returns | Description | | ------------- | ------------------------------ | ------------------------------------ | | `getPiiMap()` | `EncryptedPIIMap \| undefined` | Get the cumulative encrypted PII map | | `stats` | `object` | Get processing statistics | ### Example ```typescript theme={null} const stream = await createAnonymizerStream({ ... }); inputStream.pipe(stream).pipe(outputStream); stream.on('finish', () => { const piiMap = stream.getPiiMap(); const stats = stream.stats; console.log(`Entities: ${stats.totalEntities}, Time: ${stats.totalProcessingTimeMs}ms`); }); ``` *** ## SentenceBuffer Accumulates text and flushes at sentence boundaries with overlap for NER context. ### Constructor ```typescript theme={null} new SentenceBuffer(anonymizer: Anonymizer, config: ResolvedBufferConfig) ``` ### Methods | Method | Returns | Description | | ----------------------- | --------------------- | ------------------------------------ | | `append(chunk: string)` | `FlushResult[]` | Append text, returns flushed results | | `flush()` | `FlushResult \| null` | Force flush remaining buffer | | `getCumulativePiiMap()` | `Map` | Get all detected PII entries | | `getTotalEntities()` | `number` | Total detected entities | *** ## Types ### StreamChunkEvent ```typescript theme={null} interface StreamChunkEvent { anonymizedText: string; entities: DetectedEntity[]; totalEntities: number; processingTimeMs: number; } ``` ### StreamFinishEvent ```typescript theme={null} interface StreamFinishEvent { totalEntities: number; piiMap?: EncryptedPIIMap; totalProcessingTimeMs: number; } ``` ### FlushResult ```typescript theme={null} interface FlushResult { anonymizedText: string; entities: DetectedEntity[]; } ``` ## Related * [Streaming Guide](/guides/streaming) - Usage guide and examples * [LLM Proxy](/guides/llm-proxy) - Proxy middleware with streaming support # Anonymization & Pseudonymization Source: https://docs.rehydra.ai/concepts/anonymization-pipeline Understand Rehydra's two protection modes and the shared processing model Rehydra supports two closely related ways to protect sensitive text: `pseudonymize` for reversible workflows and `anonymize` for irreversible ones. ## The Two Modes | Mode | What happens | Can you restore originals later? | Typical use | | -------------- | ------------------------------------------------------------------------------------------ | -------------------------------- | ------------------------------------------------------- | | `pseudonymize` | Rehydra replaces detected values with placeholders and returns an encrypted `piiMap` | Yes | Apps, agents, and workflows that need later rehydration | | `anonymize` | Rehydra replaces detected values with placeholders and discards the original-value mapping | No | One-way redaction and irreversible sharing | ```typescript theme={null} import { createAnonymizer } from 'rehydra'; const reversible = createAnonymizer({ mode: 'pseudonymize' }); const irreversible = createAnonymizer({ mode: 'anonymize' }); ``` ## Shared Processing Model Both modes use the same high-level flow: Rehydra runs enabled recognizers and optional NER to identify sensitive spans. Overlaps are resolved using type priority and confidence rules. Original values are swapped for stable `` placeholders. In `pseudonymize` mode Rehydra encrypts the mapping; in `anonymize` mode it does not keep one. ## Placeholder Output Rehydra uses XML-like placeholders so downstream systems can safely process the text while preserving structure. ```xml theme={null} ``` When semantic enrichment is enabled, placeholders can include extra attributes: ```xml theme={null} ``` ## Choosing the Right Mode Use `pseudonymize` when: * You need to restore the original values later * You are building an application flow around rehydration * You want encrypted mappings for sessions or agent workflows Use `anonymize` when: * You want irreversible protection * You are preparing text for one-way sharing or analysis * You do not want to manage keys or stored mappings ## Related Concepts `pseudonymize` naturally connects to rehydration, encryption, and key management. `anonymize` is simpler, but it still depends on the same recognizers, type system, and configuration model. ## Next Steps Learn how reversible workflows restore original values. See which settings shape detection and output behavior. # Configuration Source: https://docs.rehydra.ai/concepts/configuration How Rehydra settings shape detection, output, and runtime behavior Rehydra configuration controls what gets detected, how text is protected, and whether the workflow is reversible. ## Main Configuration Areas ### Mode Choose whether the workflow is reversible: * `pseudonymize` for encrypted mappings and later rehydration * `anonymize` for irreversible protection ### Detection Control which detectors are active: * Built-in regex recognizers * Optional NER model * Custom recognizers and custom ID patterns ### Policy Policies shape detection behavior for each call or as a default: * Enabled regex and NER types * Confidence thresholds * Type priority for overlaps * Allowlists and denylists * ID reuse behavior * Leak scanning and semantic masking * Location scope exclusions (skip countries/regions) ### Tag Format By default, PII placeholders use XML-style tags: ``. You can customize the delimiters and keyword: * `tagFormat.open` / `tagFormat.close` — delimiter characters (e.g., `[[` / `]]`) * `tagFormat.keyword` — the keyword inside tags (default: `PII`) This is useful when XML-style tags conflict with your processing pipeline (e.g., HTML sanitizers, XML parsers) or when you need a format that's less likely to be mangled by translation services. ### Runtime Integrations Configuration also covers runtime behavior such as: * Key providers for reversible workflows * Storage providers for sessions * NER model backend and download settings * Semantic enrichment options ## Example ```typescript theme={null} import { createAnonymizer, PIIType } from 'rehydra'; const anonymizer = createAnonymizer({ mode: 'pseudonymize', ner: { mode: 'quantized' }, semantic: { enabled: false }, tagFormat: { open: '[[', close: ']]' }, // optional: bracket-style placeholders defaultPolicy: { regexEnabledTypes: new Set([PIIType.EMAIL, PIIType.PHONE]), nerEnabledTypes: new Set([PIIType.PERSON]), enableLeakScan: true, } }); ``` ## Configuration Levels Rehydra settings typically live at two levels: * `createAnonymizer()` configuration for reusable defaults * Per-call policy overrides when a specific operation needs different rules This lets you keep a stable baseline while still adapting behavior for individual workflows. ## Keep Concepts Separate It helps to think about configuration in layers: * Mode decides reversibility * Recognizers decide what can be found * Policy decides what should be found right now * Tag format decides how placeholders look in output * Key and storage settings decide how reversible data is managed ## Next Steps Review the full SDK configuration surface. Understand the detection layer you are configuring. # Encryption & Security Source: https://docs.rehydra.ai/concepts/encryption How Rehydra secures PII data with AES-256-GCM encryption The PII map containing original values is encrypted using **AES-256-GCM** via the Web Crypto API, which works in both Node.js and browsers. ## How It Works When you anonymize text, Rehydra: 1. Builds a mapping of placeholder IDs to original values 2. Serializes the map to JSON 3. Encrypts using AES-256-GCM with a random IV 4. Returns the encrypted bundle ```typescript theme={null} const result = await anonymizer.anonymize('Contact john@example.com'); // The encrypted PII map result.piiMap = { ciphertext: "a3f2...", // Base64-encoded encrypted data iv: "7d8c...", // Base64-encoded initialization vector authTag: "9e1f..." // Base64-encoded authentication tag }; ``` ## Key Providers Rehydra uses a `KeyProvider` interface for flexible key management: ### InMemoryKeyProvider (Development) Generates a random key on instantiation. Key is lost on page refresh. ```typescript theme={null} import { InMemoryKeyProvider } from 'rehydra'; const keyProvider = new InMemoryKeyProvider(); const anonymizer = createAnonymizer({ keyProvider }); ``` Only use `InMemoryKeyProvider` for development/testing. The key is lost when the process ends. ### ConfigKeyProvider (Production) Uses a pre-configured key from your environment: ```typescript theme={null} import { ConfigKeyProvider } from 'rehydra'; // Generate key: openssl rand -base64 32 const keyBase64 = process.env.PII_ENCRYPTION_KEY; const keyProvider = new ConfigKeyProvider(keyBase64); const anonymizer = createAnonymizer({ keyProvider }); ``` ### Custom KeyProvider Implement the interface for custom storage: ```typescript theme={null} import { KeyProvider } from 'rehydra'; class SecureKeyProvider implements KeyProvider { async getKey(): Promise { // Retrieve from secure storage, HSM, keychain, etc. return await getKeyFromSecureStorage(); } } ``` ## Decryption and Rehydration To restore original values, decrypt the PII map and rehydrate: ```typescript theme={null} import { decryptPIIMap, rehydrate } from 'rehydra'; // Get the key (must be the same key used for encryption) const key = await keyProvider.getKey(); // Decrypt the PII map const piiMap = await decryptPIIMap(result.piiMap, key); // Returns Map where key is "EMAIL:1" and value is "john@example.com" // Replace placeholders with original values const original = rehydrate(translatedText, piiMap); ``` ## Key Generation Generate a secure 256-bit key: ```typescript theme={null} import { generateKey } from 'rehydra'; // Generate random key (returns Uint8Array) const key = generateKey(); ``` Or via command line: ```bash theme={null} # Generate base64-encoded key for ConfigKeyProvider openssl rand -base64 32 ``` ## Key Derivation Derive a key from a password: ```typescript theme={null} import { deriveKey, generateSalt } from 'rehydra'; const password = 'user-password'; const salt = generateSalt(); // Store this with the encrypted data const key = await deriveKey(password, salt); ``` ## Security Best Practices Always use the encrypted storage. Never log or expose the decrypted `piiMap`. ```typescript theme={null} // ❌ Never do this console.log(piiMap); // ✅ Safe to log console.log(result.piiMap); // Only encrypted data ``` Use platform-specific secure storage: * **iOS**: Keychain * **Android**: Keystore * **Desktop**: OS keychain (via `keytar` or similar) * **Server**: Environment variables, secrets manager For long-running applications, rotate keys periodically: ```typescript theme={null} // 1. Decrypt with old key const piiMap = await decryptPIIMap(oldEncrypted, oldKey); // 2. Re-encrypt with new key const newEncrypted = await encryptPIIMap(piiMap, newKey); // 3. Store new encrypted data await storage.save(sessionId, newEncrypted); ``` Catch any missed PII in output: ```typescript theme={null} const anonymizer = createAnonymizer({ defaultPolicy: { enableLeakScan: true } }); // Check result if (!result.stats.leakScanPassed) { console.warn('Potential PII leak detected'); } ``` ## Crypto Utilities Additional cryptographic utilities: ```typescript theme={null} import { uint8ArrayToBase64, base64ToUint8Array, validateKey, secureCompare } from 'rehydra'; // Convert key to base64 for storage const keyBase64 = uint8ArrayToBase64(key); // Restore key from base64 const key = base64ToUint8Array(keyBase64); // Validate key format and length if (!validateKey(key)) { throw new Error('Invalid encryption key'); } // Constant-time comparison (prevents timing attacks) if (secureCompare(key1, key2)) { console.log('Keys match'); } ``` ## Next Steps Persist encrypted PII maps across requests Complete crypto API documentation # PII Types Source: https://docs.rehydra.ai/concepts/pii-types Understand the categories of sensitive data Rehydra can detect Rehydra detects different kinds of sensitive values through a combination of regex recognizers, optional NER, and custom patterns. ## Detection Categories ### Structured PII Structured PII has recognizable formats and is usually detected with regex-based recognizers. Common examples include: * `EMAIL` * `PHONE` * `DATE` * `IBAN` * `BIC_SWIFT` * `CREDIT_CARD` * `IP_ADDRESS` * `URL` * `ACCOUNT_NUMBER` * `TAX_ID` * `NATIONAL_ID` ### Contextual PII Contextual or "soft" PII depends on meaning and surrounding text, so it uses the optional NER model. Common examples include: * `PERSON` * `ORG` * `LOCATION` * `ADDRESS` * `DATE_OF_BIRTH` ### Secrets Secrets are credentials and keys that should never leave your local environment. Detection is **opt-in** via the `secrets` configuration. Supported types: * `API_KEY` — provider-specific keys (OpenAI, Anthropic, GitHub, Stripe, Slack, and others) * `PRIVATE_KEY` — PEM-encoded private keys (RSA, EC, DSA, OpenSSH, PGP) * `JWT` — JSON Web Tokens * `CONNECTION_STRING` — database/service URIs with embedded credentials (PostgreSQL, MySQL, MongoDB, Redis, AMQP) * `AWS_CREDENTIALS` — AWS access key IDs and secret access keys * `ENV_VAR_SECRET` — secret values in `.env`-style `KEY=VALUE` lines * `CONFIG_SECRET` — secret values in JSON, YAML, or TOML config files ```typescript theme={null} import { createAnonymizer } from 'rehydra'; const anonymizer = createAnonymizer({ secrets: { enabled: true }, }); ``` ### Domain-Specific Identifiers Some teams need to protect internal identifiers that are specific to their systems. Typical examples include: * `CASE_ID` * `CUSTOMER_ID` * Order numbers * Employee IDs * Support ticket IDs These are usually added through custom recognizers or policy-level custom patterns. ## Where Types Come From | Source | Best for | Examples | | ---------------------------- | ----------------------------- | ------------------------------------------------ | | Built-in regex recognizers | Stable, well-formed formats | Email, phone, date, IBAN, credit card | | Optional NER model | Meaning-dependent entities | Person, organization, location, address | | Secrets recognizers (opt-in) | Credentials and keys | API keys, JWTs, connection strings, private keys | | Custom recognizers | Business-specific identifiers | Case IDs, customer IDs, proprietary formats | ## Type Selection and Policy You do not have to detect every supported type on every call. Rehydra policies let you narrow detection to the types that matter for a given workflow. ```typescript theme={null} import { createAnonymizer, PIIType } from 'rehydra'; const anonymizer = createAnonymizer({ defaultPolicy: { regexEnabledTypes: new Set([PIIType.EMAIL, PIIType.PHONE]), nerEnabledTypes: new Set([PIIType.PERSON]), } }); ``` ## Overlaps and Priority Some values can match more than one detector. Rehydra resolves overlaps using the configured type priority and confidence rules so only one placeholder is emitted for a given span. ## Placeholder Shape Types appear in the placeholder output and in entity metadata: ```xml theme={null} ``` ## Next Steps See how Rehydra finds structured, contextual, and custom PII. Enable detection for API keys, tokens, and credentials. Add your own domain-specific patterns. # Recognizers Source: https://docs.rehydra.ai/concepts/recognizers How Rehydra detects sensitive values across built-in, NER, and custom sources Recognizers are the detection building blocks in Rehydra. They decide which spans count as sensitive before placeholders are generated. ## Four Recognizer Sources ### Built-in Regex Recognizers These cover structured values with stable formats, such as emails, phone numbers, dates, IBANs, URLs, and credit card numbers. They are fast, local, and usually the default foundation of a Rehydra setup. ### NER-Based Detection The optional NER model handles contextual entities that are hard to capture with regex alone, especially: * `PERSON` * `ORG` * `LOCATION` * `ADDRESS` * `DATE_OF_BIRTH` This layer is useful when privacy depends on names and other meaning-dependent entities, not just formatted identifiers. ### Secrets Recognizers The opt-in secrets recognizers detect credentials and keys such as API keys, private keys, JWTs, database connection strings, AWS credentials, and secrets embedded in `.env` files or config files. They are enabled by setting `secrets: { enabled: true }` in `createAnonymizer()`. ### Custom Recognizers Custom recognizers let you add business-specific patterns like order IDs, support tickets, claim numbers, or internal account references. They are the right tool when your domain has sensitive identifiers the built-in recognizers do not know about. ## How Recognizers Work Together Rehydra can combine multiple sources in one pass: * Regex recognizers detect structured values * NER detects contextual values if enabled * Secrets recognizers detect credentials and keys if enabled * Custom recognizers add product- or company-specific coverage * Overlapping matches are resolved by policy rules ## Registry and Extensibility The SDK exposes a recognizer registry so you can register or remove recognizers and even supply a custom registry to `createAnonymizer()`. ```typescript theme={null} const registry = anonymizer.getRegistry(); registry.register(myRecognizer); ``` ## Choosing the Right Recognizer Strategy Use built-in regex recognizers when: * Formats are predictable * You want fast local detection * You do not need model downloads Use NER when: * You need names, organizations, or locations * Context matters more than fixed format Use secrets recognizers when: * Your text may contain API keys, tokens, or database credentials * You process `.env` files or config files with embedded secrets * You want to prevent credential leaks to LLMs or external services Use custom recognizers when: * You need to protect internal identifiers * Your domain uses proprietary patterns * You want narrow, high-precision rules for business data ## Next Steps See the categories recognizers map into. Enable detection for credentials and keys. Implement your own recognizers and registry setup. # Rehydration Source: https://docs.rehydra.ai/concepts/rehydration How Rehydra restores original values in reversible workflows Rehydration is the step that turns placeholder-based text back into its original values. It only works when the text was produced in `pseudonymize` mode and you still have the matching encryption key. ## When Rehydration Is Possible Rehydration requires all of the following: * The text was processed in `pseudonymize` mode * You kept the returned encrypted `piiMap` * You still have the correct key to decrypt that `piiMap` If any of these are missing, rehydration cannot restore the originals. ## Basic Flow ```typescript theme={null} import { decryptPIIMap, rehydrate } from 'rehydra'; const key = await keyProvider.getKey(); const piiMap = await decryptPIIMap(result.piiMap, key); const restored = rehydrate(textWithPlaceholders, piiMap); ``` ## What Rehydration Uses Rehydra matches placeholders such as `` (or custom formats like `[[PII type="EMAIL" id="1"]]`) against the decrypted map and replaces them with the original values. This means the placeholder structure must stay intact between protection and restoration. If you used a custom `tagFormat` during anonymization, pass the same format to `rehydrate()`. ## Rehydration vs Anonymization | Workflow | `piiMap` exists | Originals recoverable | | -------------- | --------------- | --------------------- | | `pseudonymize` | Yes | Yes | | `anonymize` | No | No | ## Common Failure Modes * Using the wrong key * Losing the encrypted `piiMap` * Running in `anonymize` mode instead of `pseudonymize` * Modifying placeholder tags so they no longer match the stored mapping ## Where It Shows Up Rehydration is part of: * SDK workflows using `decryptPIIMap()` and `rehydrate()` * CLI workflows using `rehydra rehydrate` * Agent and plugin flows that restore values locally before tool execution ## Next Steps Learn how `piiMap` data and keys are handled. See how reversible and irreversible modes differ. # Browser Usage Source: https://docs.rehydra.ai/guides/browser-usage Using Rehydra in web browsers Rehydra works seamlessly in browsers without special configuration. This guide covers browser-specific considerations. ## Basic Example ```html theme={null} PII Anonymization ``` ## Bundler Setup ### Vite No special configuration needed: ```typescript theme={null} // vite.config.ts import { defineConfig } from 'vite'; export default defineConfig({ // Standard config - Rehydra works out of the box }); ``` ```typescript theme={null} // Your app code import { createAnonymizer } from 'rehydra'; // Automatically uses browser-safe build ``` ### Webpack Webpack 5+ handles conditional exports automatically: ```javascript theme={null} // webpack.config.js module.exports = { // Standard config - Rehydra works out of the box }; ``` ### esbuild ```bash theme={null} esbuild src/index.ts --bundle --outdir=dist --format=esm # Browser build is automatically selected ``` ## First-Use Downloads On first initialization, Rehydra downloads: | Data | Size | Storage | | --------------------- | -------- | --------- | | NER Model (quantized) | \~280 MB | OPFS | | Semantic Data | \~4 MB | IndexedDB | Show download progress to users: ```typescript theme={null} const anonymizer = createAnonymizer({ ner: { mode: 'quantized', onStatus: (status) => { document.getElementById('status').textContent = status; }, onDownloadProgress: (progress) => { const percent = Math.round(progress.percent); document.getElementById('progress').style.width = `${percent}%`; } } }); ``` ## Storage in Browsers ### Model Storage (OPFS) Large model files use the [Origin Private File System](https://developer.mozilla.org/en-US/docs/Web/API/File_System_API/Origin_private_file_system): * Persistent across sessions * Not visible in DevTools * Quota managed by browser * Cleared with "Clear site data" ### PII Map Storage (IndexedDB) For session-based storage: ```typescript theme={null} import { createAnonymizer, IndexedDBPIIStorageProvider, InMemoryKeyProvider, } from 'rehydra'; const storage = new IndexedDBPIIStorageProvider('my-app-pii'); const anonymizer = createAnonymizer({ ner: { mode: 'quantized' }, keyProvider: new InMemoryKeyProvider(), piiStorageProvider: storage, }); ``` ## ONNX Runtime The NER model requires `onnxruntime-web`. It's loaded automatically: ```typescript theme={null} // Automatic: CDN fallback if not bundled const anonymizer = createAnonymizer({ ner: { mode: 'quantized' } }); ``` For explicit bundling: ```bash theme={null} npm install onnxruntime-web ``` ```typescript theme={null} // Will use bundled version import 'onnxruntime-web'; import { createAnonymizer } from 'rehydra'; ``` ## Browser Compatibility | Browser | Minimum Version | Notes | | ------- | --------------- | ------------ | | Chrome | 86+ | Full support | | Edge | 86+ | Full support | | Firefox | 89+ | Full support | | Safari | 15.4+ | Full support | Required APIs: * [Web Crypto API](https://developer.mozilla.org/en-US/docs/Web/API/Web_Crypto_API) * [IndexedDB](https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API) * [OPFS](https://developer.mozilla.org/en-US/docs/Web/API/File_System_API/Origin_private_file_system) * [WebAssembly](https://developer.mozilla.org/en-US/docs/WebAssembly) ## Offline Support After initial download, everything works offline: ```typescript theme={null} // First visit: downloads ~290 MB await anonymizer.initialize(); // Subsequent visits: instant (data cached) await anonymizer.initialize(); // ~100ms ``` ## Memory Considerations The NER model uses \~200-300 MB RAM when loaded. For memory-constrained environments: ```typescript theme={null} // Dispose when not needed await anonymizer.dispose(); // Or use regex-only mode import { anonymizeRegexOnly } from 'rehydra'; const result = await anonymizeRegexOnly(text); // ~5 MB RAM ``` ## Web Workers For non-blocking UI, run Rehydra in a Web Worker: ```typescript theme={null} // worker.ts import { createAnonymizer } from 'rehydra'; let anonymizer; self.onmessage = async (e) => { if (e.data.type === 'init') { anonymizer = createAnonymizer({ ner: { mode: 'quantized' } }); await anonymizer.initialize(); self.postMessage({ type: 'ready' }); } if (e.data.type === 'anonymize') { const result = await anonymizer.anonymize(e.data.text); self.postMessage({ type: 'result', result }); } }; // main.ts const worker = new Worker('./worker.ts', { type: 'module' }); worker.postMessage({ type: 'init' }); ``` ## CSP Considerations If using Content Security Policy, ensure these are allowed: ``` script-src 'wasm-unsafe-eval'; # For ONNX WebAssembly connect-src https://huggingface.co; # For model downloads ``` ## What's NOT Available in Browser The browser build excludes: * `SQLitePIIStorageProvider` (use `IndexedDBPIIStorageProvider`) * Streaming anonymization (`rehydra/streaming`) * LLM proxy middleware (`rehydra/proxy`) * Node.js `fs`, `path`, `os` modules * Direct file system access ## Next Steps IndexedDB storage for browsers Bundler-specific setup # CLI Tool Source: https://docs.rehydra.ai/guides/cli Anonymize, rehydrate, and inspect PII from the terminal The Rehydra CLI lets you anonymize files and piped text directly from the terminal — no code required. ## Installation The CLI is published as a separate lightweight package: ```bash npm theme={null} npm install -g @rehydra/cli ``` ```bash bun theme={null} bun add -g @rehydra/cli ``` ```bash pnpm theme={null} pnpm add -g @rehydra/cli ``` Or run directly without installing globally: ```bash npm theme={null} npx @rehydra/cli anonymize input.txt ``` ```bash bun theme={null} bunx @rehydra/cli anonymize input.txt ``` ```bash pnpm theme={null} pnpm dlx @rehydra/cli anonymize input.txt ``` The CLI requires Node.js 18+ and depends on the `rehydra` SDK package (installed automatically). ## Commands ### anonymize Detect and replace PII in a file or stdin. ```bash theme={null} # Anonymize a file rehydra anonymize input.txt -o output.txt # Pipe from stdin echo "Contact john@example.com" | rehydra anonymize # With NER (detects names, orgs, locations) rehydra anonymize input.txt --ner quantized # JSON output rehydra anonymize input.txt -f json # Only detect specific types rehydra anonymize input.txt --types EMAIL,PHONE,IBAN # Irreversible mode (no PII map saved) rehydra anonymize input.txt --mode anonymize ``` By default, the anonymize command: * Uses regex-only detection (no NER) * Runs in `pseudonymize` mode (saves an encrypted PII map for later rehydration) * Writes the PII map to `.rehydra-pii-map.json` * Generates a random encryption key and stores it in the PII map file ### rehydrate Restore original PII values from anonymized text using the saved PII map. ```bash theme={null} # Rehydrate using the default PII map file rehydra rehydrate anonymized.txt # Specify a PII map file rehydra rehydrate anonymized.txt --pii-map my-pii-map.json # Use an explicit key (if not stored in the PII map file) rehydra rehydrate anonymized.txt --key ``` The encryption key is resolved in order: `--key` flag, `REHYDRA_KEY` environment variable, then the key embedded in the PII map file. ### inspect Dry-run that shows detected PII highlighted in context without anonymizing. Useful for previewing what will be detected. ```bash theme={null} # Inspect a file rehydra inspect input.txt # With NER rehydra inspect input.txt --ner quantized ``` Output shows the original text with PII highlighted and labeled by type using color coding. ### proxy Start a local HTTP proxy that anonymizes PII in requests to LLM APIs and rehydrates PII in responses. ```bash theme={null} # Start proxy for Anthropic (Claude Code, Cursor, etc.) rehydra proxy anthropic # Start proxy for OpenAI rehydra proxy openai # "claude" is an alias for "anthropic" rehydra proxy claude # Custom port rehydra proxy anthropic -p 9000 # Inject an API key (bridges OAuth clients to API key auth) rehydra proxy anthropic --api-key sk-ant-... # Or set the key via environment variable export LLM_API_KEY=sk-ant-... rehydra proxy anthropic ``` **Supported providers:** `openai`, `anthropic`, `claude` (alias for `anthropic`) #### How It Works 1. The proxy starts immediately with **regex-only** detection so there's no startup delay 2. In the background, the NER model downloads (if needed) and loads 3. Once NER is ready, the proxy **hot-swaps** to regex + NER detection — no restart required The `--ner` flag defaults to `quantized` for the proxy command (unlike other commands which default to `disabled`). Pass `--ner disabled` to use regex-only detection. #### Connecting Your Tools The proxy prints connection instructions on startup. Here are the common setups: ```bash Claude Code theme={null} ANTHROPIC_BASE_URL=http://127.0.0.1:8787 claude ``` ```bash Cursor (Anthropic) theme={null} # Settings → Models → Anthropic → Override Base URL # http://127.0.0.1:8787 ``` ```bash OpenAI SDK theme={null} export OPENAI_BASE_URL=http://127.0.0.1:8787/v1 # or: new OpenAI({ baseURL: "http://127.0.0.1:8787/v1" }) ``` ```bash Cursor (OpenAI) theme={null} # Settings → Models → OpenAI → Override Base URL # http://127.0.0.1:8787/v1 ``` #### API Key Injection Use `--api-key` (or the `LLM_API_KEY` environment variable) to inject an API key into upstream requests. This is useful when your tool authenticates via OAuth but the upstream API requires an API key: ```bash theme={null} # Claude Code with Max uses OAuth — bridge it to API key auth rehydra proxy claude --api-key sk-ant-... ANTHROPIC_BASE_URL=http://127.0.0.1:8787 claude ``` The proxy uses in-memory storage — PII maps are not persisted to disk. Each proxy session starts fresh. ### setup-ner Download the NER model ahead of time so anonymize/inspect don't need to download on first use. ```bash theme={null} # Download quantized model (~280 MB) rehydra setup-ner # Download standard model (~1.1 GB) rehydra setup-ner --ner standard ``` ## Round-Trip Example ```bash theme={null} # 1. Anonymize rehydra anonymize letter.txt -o anonymized.txt --ner quantized # 2. Process the anonymized text (e.g., translate externally) # ... PII placeholders are preserved by most tools ... # 3. Rehydrate rehydra rehydrate translated.txt -o final.txt ``` ## Options Reference | Flag | Short | Default | Description | | --------------------- | ----- | ----------------------- | ------------------------------------------------------------------------ | | `--output ` | `-o` | stdout | Output file | | `--format ` | `-f` | `text` | Output format: `text`, `json`, `ndjson` | | `--ner ` | | `disabled` | NER mode: `disabled`, `quantized`, `standard` | | `--pii-map ` | | `.rehydra-pii-map.json` | PII map file path | | `--key ` | | | Encryption key (base64), or set `REHYDRA_KEY` env var | | `--types ` | | all | Comma-separated PII types to detect | | `--mode ` | | `pseudonymize` | `pseudonymize` (reversible) or `anonymize` (irreversible) | | `--locale ` | | | Locale hint (e.g., `de-DE`) | | `--secrets` | | | Enable secrets/credentials detection | | `--env-file ` | | | `.env` file path for literal value redaction | | `--port ` | `-p` | `8787` | Proxy port (`proxy` command) | | `--upstream ` | | | Custom upstream URL, overrides provider default (`proxy` command) | | `--api-key ` | | | LLM API key for upstream, or set `LLM_API_KEY` env var (`proxy` command) | | `--tag-open ` | | `<` | Tag open delimiter | | `--tag-close ` | | `/>` | Tag close delimiter | | `--tag-keyword ` | | `PII` | Tag keyword | | `--no-color` | | | Disable colored output | | `--verbose` | | | Show detection statistics | | `--quiet` | `-q` | | Suppress non-essential output | | `--help` | `-h` | | Show help | | `--version` | `-V` | | Show version | ## Custom Tag Format By default, placeholders use XML-style tags: ``. Use `--tag-open`, `--tag-close`, and `--tag-keyword` to change the format: ```bash theme={null} # Bracket-style: [[PII type="EMAIL" id="1"]] rehydra anonymize input.txt --tag-open '[[' --tag-close ']]' # Custom keyword: {{REDACTED type="EMAIL" id="1"}} rehydra anonymize input.txt --tag-open '{{' --tag-close '}}' --tag-keyword REDACTED ``` The same flags work with `rehydrate`, `inspect`, and `proxy`. Use matching flags during rehydration: ```bash theme={null} rehydra anonymize input.txt -o anon.txt --tag-open '[[' --tag-close ']]' rehydra rehydrate anon.txt --tag-open '[[' --tag-close ']]' ``` ## Output Formats ### text (default) Returns the anonymized text as-is: ``` Contact or call . ``` ### json Structured JSON with anonymized text, entities, and stats: ```json theme={null} { "anonymizedText": "Contact ...", "entities": [ { "type": "EMAIL", "id": 1, "confidence": 0.95, "source": "REGEX" } ], "stats": { "totalEntities": 1, "countsByType": { "EMAIL": 1 }, "processingTimeMs": 5 } } ``` ### ndjson One JSON line per entity, plus a summary line — useful for streaming pipelines: ``` {"type":"EMAIL","id":1,"confidence":0.95,"source":"REGEX"} {"_type":"summary","anonymizedText":"...","totalEntities":1,"processingTimeMs":5} ``` ## Encryption Key Management By default, `rehydra anonymize` generates a random key and embeds it in the PII map file. For production use, supply your own key: ```bash theme={null} # Generate a key openssl rand -base64 32 # Use via environment variable export REHYDRA_KEY="your-base64-key" rehydra anonymize input.txt # Or via flag (key is NOT stored in the PII map file) rehydra anonymize input.txt --key "your-base64-key" rehydra rehydrate output.txt --key "your-base64-key" ``` When you provide a key via `--key` or `REHYDRA_KEY`, it is **not** written to the PII map file, keeping it separate from the encrypted data. ## Exit Codes | Code | Meaning | | ---- | --------------------------------- | | `0` | Success (PII found and processed) | | `1` | Error | | `2` | Success but no PII detected | ## Next Steps Use Rehydra programmatically in your app See all PII types the CLI can detect # Custom Recognizers Source: https://docs.rehydra.ai/guides/custom-recognizers Add domain-specific PII detection patterns Extend Rehydra with custom recognizers for domain-specific identifiers like order numbers, employee IDs, or proprietary formats. ## Quick Start ```typescript theme={null} import { createAnonymizer, createCustomIdRecognizer, PIIType } from 'rehydra'; // Create a custom recognizer const orderRecognizer = createCustomIdRecognizer([ { name: 'Order Number', pattern: /\bORD-[A-Z0-9]{8}\b/g, type: PIIType.CASE_ID, }, ]); // Register it const anonymizer = createAnonymizer(); anonymizer.getRegistry().register(orderRecognizer); await anonymizer.initialize(); // Use it const result = await anonymizer.anonymize('Your order ORD-ABC12345 is shipped'); // "Your order is shipped" ``` ## Custom ID Recognizer For simple pattern-based detection, use `createCustomIdRecognizer`: ```typescript theme={null} import { createCustomIdRecognizer, PIIType } from 'rehydra'; const recognizer = createCustomIdRecognizer([ { name: 'Order Number', // Descriptive name pattern: /\bORD-[A-Z0-9]{8}\b/g, // Regex pattern (must have 'g' flag) type: PIIType.CASE_ID, // PII type to assign }, { name: 'Employee ID', pattern: /\bEMP-[0-9]{6}\b/g, type: PIIType.CUSTOMER_ID, }, { name: 'Support Ticket', pattern: /\bTICKET-[0-9]+\b/gi, // Case-insensitive type: PIIType.CASE_ID, }, ]); ``` Always include the `g` (global) flag in your regex patterns. Without it, only the first match will be found. ## Available PII Types for Custom Patterns | Type | Use Case | | ------------------------ | -------------------------------------------- | | `PIIType.CASE_ID` | Order numbers, ticket IDs, reference numbers | | `PIIType.CUSTOMER_ID` | Customer IDs, employee IDs, account numbers | | `PIIType.ACCOUNT_NUMBER` | Financial account numbers | | `PIIType.TAX_ID` | Tax identification numbers | | `PIIType.NATIONAL_ID` | National ID numbers, SSN, etc. | ## Extending RegexRecognizer For more control, extend the `RegexRecognizer` class: ```typescript theme={null} import { RegexRecognizer, PIIType, SpanMatch } from 'rehydra'; class MyCustomRecognizer extends RegexRecognizer { constructor() { super( 'MyCustomRecognizer', // Name PIIType.CASE_ID, // PII type /\bMY-[A-Z0-9]+\b/g, // Pattern 1.0 // Confidence (0.0-1.0) ); } // Override to add validation protected validate(match: string): boolean { // Add custom validation logic return match.length >= 6 && match.length <= 20; } // Override to modify confidence based on context protected getConfidence(match: string): number { // Higher confidence for longer matches return match.length > 10 ? 1.0 : 0.9; } } const anonymizer = createAnonymizer(); anonymizer.getRegistry().register(new MyCustomRecognizer()); ``` ## Built-in Recognizers Rehydra includes these recognizers by default: | Recognizer | PII Type | Pattern Examples | | ---------------------- | ------------ | ------------------------------------------- | | `emailRecognizer` | EMAIL | [user@example.com](mailto:user@example.com) | | `phoneRecognizer` | PHONE | +49 30 123456 | | `ibanRecognizer` | IBAN | DE89370400440532013000 | | `bicSwiftRecognizer` | BIC\_SWIFT | COBADEFFXXX | | `creditCardRecognizer` | CREDIT\_CARD | 4111111111111111 | | `ipAddressRecognizer` | IP\_ADDRESS | 192.168.1.1, ::1 | | `urlRecognizer` | URL | [https://example.com](https://example.com) | ### Secrets Recognizers (opt-in) These are registered automatically when `secrets: { enabled: true }` is set: | Recognizer | PII Type | Pattern Examples | | ---------------------------- | ------------------ | -------------------------------------- | | `apiKeyRecognizer` | API\_KEY | `sk-proj-...`, `sk-ant-...`, `ghp_...` | | `privateKeyRecognizer` | PRIVATE\_KEY | `-----BEGIN RSA PRIVATE KEY-----` | | `jwtRecognizer` | JWT | `eyJhbGciOiJIUzI1NiJ9...` | | `connectionStringRecognizer` | CONNECTION\_STRING | `postgres://user:pass@host/db` | | `awsCredentialsRecognizer` | AWS\_CREDENTIALS | `AKIA...` | | `envVarSecretRecognizer` | ENV\_VAR\_SECRET | `API_KEY=sk-abc123` | | `configSecretRecognizer` | CONFIG\_SECRET | `"database_password": "s3cret"` | See the [Secrets Detection guide](/guides/secrets-detection) for full configuration options. ## Recognizer Registry Manage recognizers through the registry: ```typescript theme={null} const registry = anonymizer.getRegistry(); // Register a recognizer registry.register(myRecognizer); // List all recognizers const recognizers = registry.getAll(); // Get specific recognizer by name const email = registry.get('EmailRecognizer'); // Remove a recognizer registry.unregister('MyCustomRecognizer'); ``` ## Create a New Registry For complete control, create a custom registry: ```typescript theme={null} import { createRegistry, emailRecognizer, phoneRecognizer } from 'rehydra'; // Start with empty registry const registry = createRegistry(); // Add only the recognizers you need registry.register(emailRecognizer); registry.register(phoneRecognizer); registry.register(myCustomRecognizer); // Use custom registry const anonymizer = createAnonymizer({ registry }); ``` ## Disabling Built-in Recognizers To disable specific built-in recognizers: ```typescript theme={null} import { createAnonymizer, PIIType } from 'rehydra'; const anonymizer = createAnonymizer({ defaultPolicy: { // Only enable specific regex types regexEnabledTypes: new Set([ PIIType.EMAIL, PIIType.PHONE, // Exclude IBAN, CREDIT_CARD, etc. ]), } }); ``` ## Pattern Best Practices Prevent matching substrings: ```typescript theme={null} // ❌ Bad: matches "ABCORD-12345XYZ" pattern: /ORD-[0-9]+/g // ✅ Good: only matches complete pattern pattern: /\bORD-[0-9]+\b/g ``` Avoid overly broad patterns: ```typescript theme={null} // ❌ Too broad: matches any numbers pattern: /[0-9]+/g // ✅ Specific: matches expected format pattern: /\bORD-[0-9]{6,10}\b/g ``` Validate matches for accuracy: ```typescript theme={null} class ValidatedRecognizer extends RegexRecognizer { protected validate(match: string): boolean { // Check format if (!match.startsWith('ORD-')) return false; // Check length if (match.length < 10) return false; // Check checksum if applicable return this.validateChecksum(match); } } ``` Complex patterns can slow detection: ```typescript theme={null} // ❌ Slow: complex lookahead/lookbehind pattern: /(?<=ID:)\s*[A-Z]+(?=\s)/g // ✅ Fast: simple pattern + post-processing pattern: /ID:\s*[A-Z]+/g // Then strip "ID:" in validate() ``` ## Testing Recognizers ```typescript theme={null} import { describe, it, expect } from 'vitest'; import { createCustomIdRecognizer, PIIType, createAnonymizer } from 'rehydra'; describe('Order Recognizer', () => { const recognizer = createCustomIdRecognizer([ { name: 'Order', pattern: /\bORD-[A-Z0-9]{8}\b/g, type: PIIType.CASE_ID }, ]); it('detects valid order numbers', async () => { const anonymizer = createAnonymizer(); anonymizer.getRegistry().register(recognizer); await anonymizer.initialize(); const result = await anonymizer.anonymize('Order: ORD-ABC12345'); expect(result.entities).toHaveLength(1); expect(result.entities[0].type).toBe('CASE_ID'); }); it('ignores invalid patterns', async () => { const anonymizer = createAnonymizer(); anonymizer.getRegistry().register(recognizer); await anonymizer.initialize(); const result = await anonymizer.anonymize('Order: ORD-123'); // Too short expect(result.entities).toHaveLength(0); }); }); ``` ## Next Steps All supported PII types Complete API documentation # LLM Proxy Source: https://docs.rehydra.ai/guides/llm-proxy Automatic PII anonymization and rehydration for OpenAI and Anthropic API calls The LLM proxy middleware intercepts requests to LLM APIs, anonymizes PII in outgoing messages, and rehydrates PII in responses — all transparently. The proxy module is only available in Node.js and Bun. It is not included in the browser build. ## Supported Providers | Provider | Detection | Streaming | | ---------------------------- | -------------------------------------------- | --------- | | OpenAI (and compatible APIs) | URL, `Bearer sk-` header | SSE | | Anthropic | URL, `x-api-key`/`anthropic-version` headers | SSE | ## Integration Methods From simplest to most flexible: | Method | Use Case | | -------------------------------------------------------- | -------------------------------------- | | [`wrapLLMClient()`](#wrap-an-sdk-client) | Wrap an OpenAI or Anthropic SDK client | | [`createRehydraFetch()`](#custom-fetch) | Drop-in `fetch` replacement | | [`createRehydraProxy()`](#proxy-middleware) | HTTP middleware for any framework | | [`createRehydraProxyServer()`](#standalone-proxy-server) | Standalone proxy server | *** ## Wrap an SDK Client The simplest approach — wrap your existing OpenAI or Anthropic client: ```typescript theme={null} import OpenAI from 'openai'; import { wrapLLMClient } from 'rehydra/proxy'; import { InMemoryKeyProvider, SQLitePIIStorageProvider } from 'rehydra'; const storage = new SQLitePIIStorageProvider('./pii.db'); await storage.initialize(); const client = new OpenAI(); const wrappedClient = wrapLLMClient(client, { anonymizer: { ner: { mode: 'quantized' } }, keyProvider: new InMemoryKeyProvider(), piiStorageProvider: storage, getSessionId: (req) => 'session-123', }); // Use exactly as before — PII is anonymized/rehydrated automatically const response = await wrappedClient.chat.completions.create({ model: 'gpt-4', messages: [{ role: 'user', content: 'My name is John Smith and my email is john@acme.com' }], }); console.log(response.choices[0].message.content); // Response contains original PII values (rehydrated) ``` Works the same way with Anthropic: ```typescript theme={null} import Anthropic from '@anthropic-ai/sdk'; import { wrapLLMClient } from 'rehydra/proxy'; const client = new Anthropic(); const wrappedClient = wrapLLMClient(client, { anonymizer: { ner: { mode: 'quantized' } }, keyProvider: new InMemoryKeyProvider(), piiStorageProvider: storage, getSessionId: (req) => 'session-123', }); ``` *** ## Custom Fetch Replace the `fetch` function used by any SDK: ```typescript theme={null} import { createRehydraFetch } from 'rehydra/proxy'; import { InMemoryKeyProvider, SQLitePIIStorageProvider } from 'rehydra'; const storage = new SQLitePIIStorageProvider('./pii.db'); await storage.initialize(); const rehydraFetch = createRehydraFetch({ anonymizer: { ner: { mode: 'quantized' } }, keyProvider: new InMemoryKeyProvider(), piiStorageProvider: storage, getSessionId: (req) => 'session-123', }); // Use with OpenAI SDK const client = new OpenAI({ fetch: rehydraFetch }); // Or use directly const response = await rehydraFetch('https://api.openai.com/v1/chat/completions', { method: 'POST', headers: { 'Authorization': `Bearer ${apiKey}`, 'Content-Type': 'application/json' }, body: JSON.stringify({ model: 'gpt-4', messages: [{ role: 'user', content: 'Contact john@example.com' }], }), }); ``` *** ## Proxy Middleware For framework integration (Hono, Bun.serve, etc.): ```typescript theme={null} import { createRehydraProxy } from 'rehydra/proxy'; import { InMemoryKeyProvider, SQLitePIIStorageProvider } from 'rehydra'; const storage = new SQLitePIIStorageProvider('./pii.db'); await storage.initialize(); const proxy = createRehydraProxy({ upstream: 'https://api.openai.com', anonymizer: { ner: { mode: 'quantized' } }, keyProvider: new InMemoryKeyProvider(), piiStorageProvider: storage, getSessionId: (req) => 'session-123', }); // Use with Bun.serve Bun.serve({ port: 8080, fetch: proxy, }); // Or with Hono import { Hono } from 'hono'; const app = new Hono(); app.all('/v1/*', (c) => proxy(c.req.raw)); ``` Configure your LLM client to point at the proxy: ```typescript theme={null} const client = new OpenAI({ baseURL: 'http://localhost:8080/v1', }); ``` *** ## Standalone Proxy Server Start a proxy server with one function call: ```typescript theme={null} import { createRehydraProxyServer } from 'rehydra/proxy'; import { InMemoryKeyProvider, SQLitePIIStorageProvider } from 'rehydra'; const storage = new SQLitePIIStorageProvider('./pii.db'); await storage.initialize(); const server = await createRehydraProxyServer({ upstream: 'https://api.openai.com', port: 8080, anonymizer: { ner: { mode: 'quantized' } }, keyProvider: new InMemoryKeyProvider(), piiStorageProvider: storage, getSessionId: (req) => 'session-123', }); console.log(`Proxy running at http://${server.host}:${server.port}`); // Point any client at the proxy // const client = new OpenAI({ baseURL: 'http://localhost:8080/v1' }); // Later: shut down await server.close(); ``` *** ## Streaming Support All proxy methods support streaming (SSE) responses. PII is rehydrated in each streamed chunk: ```typescript theme={null} const wrappedClient = wrapLLMClient(client, { anonymizer: { ner: { mode: 'quantized' } }, keyProvider: new InMemoryKeyProvider(), piiStorageProvider: storage, getSessionId: (req) => 'session-123', }); // Streaming works transparently const stream = await wrappedClient.chat.completions.create({ model: 'gpt-4', messages: [{ role: 'user', content: 'Tell me about John Smith' }], stream: true, }); for await (const chunk of stream) { process.stdout.write(chunk.choices[0]?.delta?.content || ''); // PII values are rehydrated in each chunk } ``` ### Streaming Internals PII placeholders like `` can span SSE chunk boundaries. The proxy buffers incomplete tags across chunks and only rehydrates once a complete tag is available, so streaming rehydration is reliable regardless of how the upstream chunks its response. You can disable streaming rehydration if needed: ```typescript theme={null} const wrappedClient = wrapLLMClient(client, { keyProvider: new InMemoryKeyProvider(), piiStorageProvider: storage, handleStreaming: false, // SSE responses pass through without rehydration }); ``` ## Passthrough Behavior The proxy only intercepts **POST requests with `application/json` content type**. Everything else is forwarded unchanged: * **Non-POST requests** (GET, OPTIONS, etc.) — passed through to upstream * **Non-JSON content types** — passed through to upstream * **Non-JSON responses** — passed through with original status code This means health checks, CORS preflight, and other non-chat requests work without interference. *** ## Tool Call Handling The proxy automatically anonymizes and rehydrates tool/function call arguments. If the LLM returns a tool call whose arguments contain PII placeholders, Rehydra rehydrates them before your code sees the result. This works for both non-streaming and streaming responses: ```typescript theme={null} const wrappedClient = wrapLLMClient(client, { anonymizer: { ner: { mode: 'quantized' } }, keyProvider: new InMemoryKeyProvider(), piiStorageProvider: storage, getSessionId: () => 'session-123', }); const response = await wrappedClient.chat.completions.create({ model: 'gpt-4', messages: [{ role: 'user', content: 'Send an email to John Smith at john@acme.com' }], tools: [{ type: 'function', function: { name: 'send_email', parameters: { type: 'object', properties: { to: { type: 'string' }, name: { type: 'string' }, }, }, }, }], }); // Tool call arguments are rehydrated — original PII values restored const args = JSON.parse(response.choices[0].message.tool_calls[0].function.arguments); // { to: "john@acme.com", name: "John Smith" } ``` In streaming mode, tool call argument chunks are buffered per tool call index and rehydrated when the tool call completes. *** ## Automated Tool Execution Loop For server-side agentic workflows, the proxy can manage the full multi-round tool execution loop automatically. Provide an `onToolCall` callback and the proxy will: 1. Rehydrate tool call arguments (so your function receives real PII values) 2. Call your callback with the tool name and parsed arguments 3. Anonymize the tool result before sending it back to the LLM 4. Repeat until the LLM responds with no tool calls, or `maxToolRounds` is reached ```typescript theme={null} import OpenAI from 'openai'; import { createRehydraFetch } from 'rehydra/proxy'; import { InMemoryKeyProvider, SQLitePIIStorageProvider } from 'rehydra'; const storage = new SQLitePIIStorageProvider('./pii.db'); await storage.initialize(); const rehydraFetch = createRehydraFetch({ anonymizer: { ner: { mode: 'quantized' } }, keyProvider: new InMemoryKeyProvider(), piiStorageProvider: storage, getSessionId: () => 'session-1', // Tool execution callback — arguments are already rehydrated onToolCall: async (name, args, toolCallId) => { if (name === 'send_email') { return await sendEmail(args.to as string, args.body as string); } if (name === 'lookup_user') { return await db.users.find(args.email as string); } return { error: `Unknown tool: ${name}` }; }, maxToolRounds: 5, // default: 10 }); const client = new OpenAI({ fetch: rehydraFetch }); // The proxy handles the full tool loop — this single call may // result in multiple LLM round-trips behind the scenes const response = await client.chat.completions.create({ model: 'gpt-4', messages: [{ role: 'user', content: 'Send a welcome email to john@acme.com' }], tools: [/* your tool definitions */], }); // Final response (after all tool rounds complete) console.log(response.choices[0].message.content); ``` The automated tool loop only works with **non-streaming** requests. For streaming tool calls, handle the tool loop manually using the rehydrated arguments from each streamed response. *** ## PII System Instruction When PII is detected in an outgoing request, the proxy automatically injects a system instruction telling the LLM to preserve PII placeholder tags in its response. This prevents the model from inventing replacement values. The instruction is injected as an OpenAI system message or an Anthropic system field, depending on the provider. You can customize this behavior with the `systemInstruction` config option: ```typescript theme={null} const rehydraFetch = createRehydraFetch({ // ... // Custom instruction systemInstruction: 'Do not modify any XML tags in the conversation.', // Or disable entirely systemInstruction: false, // Or omit for the built-in default (recommended) }); ``` | Value | Behavior | | --------------------- | ------------------------------------------------------ | | `undefined` (default) | Built-in instruction injected when PII is detected | | `string` | Your custom instruction, injected when PII is detected | | `false` | No instruction injected | *** ## Session Management Use `getSessionId` to associate requests with sessions. This enables consistent entity IDs and cross-request rehydration: ```typescript theme={null} const wrappedClient = wrapLLMClient(client, { anonymizer: { ner: { mode: 'quantized' } }, keyProvider: new InMemoryKeyProvider(), piiStorageProvider: storage, getSessionId: (req) => { // Extract session from custom header, URL, or other context const url = new URL(req.url); return url.searchParams.get('session') || 'default'; }, }); ``` ## Error Handling When the proxy encounters an error, it returns a JSON response with the following shape: ```json theme={null} { "error": { "message": "description of what went wrong", "type": "rehydra_proxy_error" } } ``` | Status Code | Meaning | | ----------- | -------------------------------------------------------- | | 400 | Invalid JSON in the request body | | 500 | Internal proxy error during anonymization or rehydration | | 502 | Upstream LLM API is unreachable | | 503 | Proxy not ready (initialization failed) | *** ## Next Steps Stream-level anonymization for chunked text Complete proxy API documentation # NER Detection Source: https://docs.rehydra.ai/guides/ner-detection Configure the NER model for detecting names, organizations, and locations The NER (Named Entity Recognition) model enables detection of soft PII like person names, organizations, and locations that can't be captured by regex patterns. ## Model Modes | Mode | Description | Size | Use Case | | ------------- | ----------------------- | -------- | ------------------------------------ | | `'disabled'` | No NER, regex only | 0 | Fast processing, structured PII only | | `'quantized'` | Smaller quantized model | \~280 MB | Recommended for most use cases | | `'standard'` | Full-size model | \~1.1 GB | Maximum accuracy | | `'custom'` | Your own ONNX model | Varies | Domain-specific models | ## Basic Setup ```typescript theme={null} import { createAnonymizer } from 'rehydra'; const anonymizer = createAnonymizer({ ner: { mode: 'quantized', onStatus: (status) => console.log(status), } }); await anonymizer.initialize(); // Downloads model on first use const result = await anonymizer.anonymize('Hello John Smith from Acme Corp!'); // "Hello from !" ``` ## Download Progress Track model download progress: ```typescript theme={null} const anonymizer = createAnonymizer({ ner: { mode: 'quantized', onStatus: (status) => console.log('Status:', status), onDownloadProgress: (progress) => { console.log(`${progress.file}: ${progress.percent}%`); } } }); ``` Output during first initialization: ``` Status: Downloading model files... model.onnx: 15% model.onnx: 30% model.onnx: 100% vocab.txt: 100% Status: Loading NER model... Status: NER model loaded! ``` ## Confidence Thresholds NER entities have confidence scores (0.0-1.0). Configure minimum thresholds: ```typescript theme={null} const anonymizer = createAnonymizer({ ner: { mode: 'quantized', thresholds: { PERSON: 0.8, // 80% confidence required ORG: 0.7, // 70% for organizations LOCATION: 0.6, // 60% for locations } } }); ``` Lower thresholds → more detections (potentially more false positives) Higher thresholds → fewer detections (may miss some entities) ## Case Fallback The NER model is case-sensitive — it works best on properly capitalized text. This means lowercase names like `"tom"` or `"sarah"` can be missed. Enable `caseFallback` to run a second NER pass on title-cased text and merge any new detections: ```typescript theme={null} const anonymizer = createAnonymizer({ ner: { mode: 'quantized', caseFallback: true, } }); await anonymizer.initialize(); await anonymizer.anonymize('hey tom, can you ask sarah to call me?'); // "hey , can you ask to call me?" ``` Without `caseFallback`, neither `"tom"` nor `"sarah"` would be detected. ### How it works 1. The primary NER pass runs on the original text 2. A second pass runs on title-cased text (e.g. `"tom"` → `"Tom"`) 3. New detections from the fallback pass that don't overlap with primary detections are merged in 4. Fallback detections keep the original lowercase text and character offsets 5. A confidence penalty is applied to fallback detections to reduce false positives ### Confidence penalty Fallback detections receive a confidence penalty (multiplied by `caseFallbackPenalty`, default `0.85`) since title-casing can introduce false positives. You can tune this: ```typescript theme={null} const anonymizer = createAnonymizer({ ner: { mode: 'quantized', caseFallback: true, caseFallbackPenalty: 0.7, // Stricter penalty } }); ``` Enabling `caseFallback` doubles NER inference time since it runs two passes. Use it when your input text contains informal or uncapitalized names (chat messages, transcripts, etc.). ## Auto-Download Control By default, models are downloaded automatically. To disable: ```typescript theme={null} const anonymizer = createAnonymizer({ ner: { mode: 'quantized', autoDownload: false, // Will throw if model not present } }); ``` ## Manual Model Management Pre-download models or manage cache: ```typescript theme={null} import { isModelDownloaded, downloadModel, clearModelCache, listDownloadedModels } from 'rehydra'; // Check if model exists const hasModel = await isModelDownloaded('quantized'); // Pre-download with progress await downloadModel('quantized', (progress) => { console.log(`${progress.file}: ${progress.percent}%`); }); // List downloaded models const models = await listDownloadedModels(); // ['quantized'] // Clear specific model await clearModelCache('quantized'); // Clear all models await clearModelCache(); ``` ## Inference Server Backend For batch processing or GPU acceleration, offload NER inference to a remote server: ```typescript theme={null} const anonymizer = createAnonymizer({ ner: { mode: 'quantized', backend: 'inference-server', inferenceServerUrl: 'http://localhost:8000/predict', inferenceServerTimeout: 30000, // 30 seconds (default) } }); ``` This sends tokenized text to the server for inference instead of running ONNX locally. The server must accept the same input format and return logits in the expected shape. ## Custom Models Use your own ONNX model: ```typescript theme={null} const anonymizer = createAnonymizer({ ner: { mode: 'custom', modelPath: './my-model.onnx', vocabPath: './vocab.txt', } }); ``` Custom models must follow the same input/output format as the default models. See the [model training guide](https://github.com/rehydra-ai/rehydra-sdk#building-custom-models) for details. ## Cache Locations Models are cached locally for offline use: ### Node.js | Platform | Location | | -------- | ---------------------------------- | | macOS | `~/Library/Caches/rehydra/models/` | | Linux | `~/.cache/rehydra/models/` | | Windows | `%LOCALAPPDATA%/rehydra/models/` | ### Browser In browsers, models are stored using: * **Origin Private File System (OPFS)** for large model files * **IndexedDB** for metadata Data persists across page reloads and browser sessions. ## NER-Detected Types | Type | Examples | | --------------- | --------------------------------- | | `PERSON` | John Smith, Maria, Dr. Johnson | | `ORG` | Acme Corp, Google, United Nations | | `LOCATION` | Berlin, Germany, Central Park | | `ADDRESS` | 123 Main Street | | `DATE_OF_BIRTH` | born on March 15, 1990 | ## Disabling Specific NER Types Detect only certain entity types: ```typescript theme={null} import { createAnonymizer, PIIType } from 'rehydra'; const anonymizer = createAnonymizer({ ner: { mode: 'quantized' }, defaultPolicy: { nerEnabledTypes: new Set([ PIIType.PERSON, // Only detect names ]) } }); ``` ## Performance Tips Model loading is expensive. Create once and reuse: ```typescript theme={null} // ✅ Good: create once const anonymizer = createAnonymizer({ ner: { mode: 'quantized' } }); await anonymizer.initialize(); // Reuse for multiple texts await anonymizer.anonymize(text1); await anonymizer.anonymize(text2); // Dispose when done await anonymizer.dispose(); ``` The quantized model is \~95% as accurate but 4x smaller: | Model | Size | Inference Time | | --------- | -------- | -------------- | | Standard | \~1.1 GB | \~120ms | | Quantized | \~280 MB | \~100ms | If you only need emails, phones, IBANs, etc.: ```typescript theme={null} import { anonymizeRegexOnly } from 'rehydra'; const result = await anonymizeRegexOnly(text); // ~5ms instead of ~150ms ``` ## Next Steps Add gender and location attributes Add domain-specific detection patterns # OpenCode Plugin Source: https://docs.rehydra.ai/guides/opencode-plugin Prevent your coding agent from leaking PII and secrets to LLM providers The Rehydra OpenCode plugin intercepts the conversation between [OpenCode](https://github.com/sst/opencode) and the LLM. It automatically detects PII and secrets in all messages, replaces them with placeholders before they leave your machine, and transparently restores real values before any tool executes locally. ## Installation ```bash npm theme={null} npm install @rehydra/opencode ``` ```bash bun theme={null} bun add @rehydra/opencode ``` ```bash pnpm theme={null} pnpm add @rehydra/opencode ``` Add the plugin to your `opencode.json`: ```json theme={null} { "plugin": ["@rehydra/opencode"] } ``` The plugin automatically detects PII (emails, phone numbers, credit cards, etc.) and secrets (API keys, JWTs, connection strings, etc.) in all messages. It also reads `.env` in your project root to catch exact secret values wherever they appear. Requires Node.js 18+ and depends on the `rehydra` SDK package (installed automatically). ## Configuration For custom settings, create `.opencode/plugins/rehydra.ts`: ```typescript theme={null} import { createRehydraPlugin } from "@rehydra/opencode"; export default createRehydraPlugin({ // Scan multiple env files envFiles: [".env", ".env.local", ".env.production"], // Always redact these values, even if not in .env redactValues: ["sk-live-abc123..."], // Minimum value length to consider a secret (default: 8) minValueLength: 6, // Re-enable URL and IP_ADDRESS detection (disabled by default) disableTypes: [], }); ``` ### Options | Option | Type | Default | Description | | ---------------- | --------------------- | -------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | | `envFiles` | `string[]` | `[".env"]` | `.env` files to scan for secret values | | `redactValues` | `string[]` | `[]` | Explicit values to always redact | | `minValueLength` | `number` | `8` | Minimum value length to consider a secret | | `disableTypes` | `string[]` | `["URL", "IP_ADDRESS"]` | PII types to skip — URLs and IP addresses are disabled by default since coding agents routinely work with these values | | `locale` | `string` | | Locale hint for anonymization (e.g., `de-DE`) | | `policy` | `AnonymizationPolicy` | | Policy overrides for anonymization behavior | | `tagFormat` | `TagFormat` | `{ open: "<", close: "/>", keyword: "PII" }` | Tag format for PII placeholders (also available via `anonymizer.tagFormat`) | | `anonymizer` | `AnonymizerConfig` | | Advanced: full anonymizer config (overrides `envFiles`, `redactValues`, and `minValueLength`) | ## What Gets Detected The plugin uses Rehydra's full detection suite automatically — no configuration needed. **Structured PII** — detected via pattern matching: * Emails, phone numbers, credit card numbers * IBANs, BIC/SWIFT codes, account numbers * IP addresses, URLs (disabled by default — set `disableTypes: []` to enable) * Tax IDs, national IDs **Secrets** — detected via provider-specific patterns: * API keys (OpenAI, Anthropic, GitHub, Stripe, Slack, and others) * JWTs, private keys (PEM), connection strings * AWS access keys and secret keys * Secret values in `.env` files and JSON/YAML/TOML configs **Soft PII** (optional, via NER model — not recommended due to significant added latency): * Names, organizations, locations, addresses **Additional sources:** * Exact values from `.env` files (matched anywhere in text) * Explicit values passed via `redactValues` ## How It Works The plugin uses five OpenCode hooks — everything runs locally and no data leaves your machine except the scrubbed conversation sent to the LLM provider. | Hook | What it does | | --------------------- | --------------------------------------------------------------------------------------- | | `messages.transform` | Scrubs PII and secrets from all message text and tool outputs before they reach the LLM | | `system.transform` | Injects an instruction telling the LLM to treat placeholders as real values | | `tool.execute.before` | Restores real values in tool arguments before local execution | | `tool.execute.after` | Restores real values in displayed tool output | | `text.complete` | Restores real values in LLM response text shown to you | The system prompt instruction is only injected after at least one secret has been anonymized, keeping prompts clean when no secrets are present. ### Session Handling Each OpenCode session gets its own Rehydra session. PII mappings are consistent within a session — the same secret always maps to the same placeholder, so the LLM can reason about relationships between values without seeing the real data. ### Tool Argument Rehydration When the LLM generates a tool call (e.g., a shell command containing a placeholder), the plugin deep-walks all argument values and restores real secrets before execution. This means commands like `git push`, `curl`, or `zwrm secrets set` receive the actual credential values. OpenCode's `tool.execute.before` hook passes args by reference — the plugin mutates args in-place rather than replacing the object, which is required for the rehydrated values to take effect. ## Logging Plugin activity is logged to OpenCode's log directory (`~/.local/share/opencode/log/`). Run OpenCode with `--log-level DEBUG` for detailed output. ``` INFO service=rehydra scrubbed={"ENV_VAR_SECRET":2} messageCount=3 scrubbed 2 secret(s) from messages INFO service=rehydra tool=bash callID=call_abc123 rehydrated PII tags in tool args ``` ## Custom Tag Format Use `tagFormat` to change placeholder delimiters, for example when XML-style tags interfere with your workflow: ```typescript theme={null} import { createRehydraPlugin } from "@rehydra/opencode"; export default createRehydraPlugin({ tagFormat: { open: "[[", close: "]]" }, }); ``` This produces placeholders like `[[PII type="EMAIL" id="1"]]` instead of ``. ## Advanced: Full Anonymizer Config For full control, pass an `anonymizer` config directly. This overrides `envFiles`, `redactValues`, and `minValueLength`: ```typescript theme={null} import { createRehydraPlugin } from "@rehydra/opencode"; export default createRehydraPlugin({ anonymizer: { secrets: { enabled: true, envFiles: [".env", ".env.local"], minValueLength: 6, }, ner: { mode: "quantized", }, }, }); ``` See the [`createAnonymizer` API reference](/api-reference/create-anonymizer) for all available options. ## Next Steps See all PII types Rehydra can detect Anonymize API calls in server-side workflows # Secrets Detection Source: https://docs.rehydra.ai/guides/secrets-detection Detect API keys, tokens, connection strings, and other credentials Rehydra can detect secrets and credentials embedded in text, config files, and environment variables. Secrets detection is **opt-in** — enable it when your workflow handles content that may contain credentials. ## Enable Secrets Detection ```typescript theme={null} import { createAnonymizer } from 'rehydra'; const anonymizer = createAnonymizer({ secrets: { enabled: true }, }); await anonymizer.initialize(); const result = await anonymizer.anonymize( 'Connect with postgres://admin:s3cret@db.internal:5432/app' ); // "Connect with " ``` ## What Gets Detected Provider-specific key patterns including: * **OpenAI** — `sk-proj-*`, `sk-*` * **Anthropic** — `sk-ant-*` * **GitHub** — PATs (`ghp_*`), OAuth (`ghu_*`), App tokens (`ghs_*`) * **Stripe** — `sk_live_*`, `sk_test_*`, `rk_*` * **Slack** — `xoxb-*`, `xoxp-*`, `xoxa-*`, `xoxs-*` * **SendGrid** — `SG.*.*` * **Twilio** — `SK` followed by 32 hex characters * **Mailgun** — `key-` followed by 32 hex characters PEM-encoded private key blocks with `BEGIN`/`END` markers. Supports RSA, EC, DSA, OpenSSH, PGP, and encrypted variants. Three-segment base64url tokens starting with `eyJ`. Validates that the decoded header contains an `alg` field. Database and service URIs with embedded credentials: * PostgreSQL — `postgres://user:pass@host/db` * MySQL / MariaDB — `mysql://user:pass@host/db` * MongoDB — `mongodb+srv://user:pass@host/db` * Redis — `redis://:pass@host:port` * AMQP — `amqp://user:pass@host:port` Placeholder passwords like `changeme` or `password` are ignored. * Access key IDs with the `AKIA` prefix (high confidence) * Secret access keys when AWS-related context keywords are present (context-dependent) Secret values in `.env`-style lines like `API_KEY=sk-abc123`. The recognizer checks whether the key name suggests a secret (e.g., names containing `password`, `secret`, `token`, `api_key`, etc.) and filters out placeholder values. Secret values in JSON, YAML, and TOML configuration files. Uses the same key name heuristics as the environment variable recognizer. ```json theme={null} { "database_password": "s3cret-value" } ``` ```yaml theme={null} api_secret: sk-live-abc123 ``` ## Scanning .env Files Point Rehydra at your `.env` files so it can learn the exact secret values and detect them anywhere in the text — even outside of `KEY=VALUE` context: ```typescript theme={null} const anonymizer = createAnonymizer({ secrets: { enabled: true, envFiles: ['.env', '.env.local'], }, }); ``` This parses the files, extracts the values, and registers a literal value recognizer that matches those exact strings with maximum confidence. ## Explicit Redact Values If you know specific values that should always be redacted, pass them directly: ```typescript theme={null} const anonymizer = createAnonymizer({ secrets: { enabled: true, redactValues: ['my-production-api-key', 'db-password-here'], }, }); ``` ## Custom Key Name Patterns The built-in key name heuristic covers common names like `password`, `secret`, `token`, `api_key`, `connection_string`, and about 30 others. To extend it: ```typescript theme={null} const anonymizer = createAnonymizer({ secrets: { enabled: true, secretKeyPatterns: [ /^INTERNAL_.*_KEY$/i, /^MY_APP_SECRET_/i, ], }, }); ``` These extra patterns apply to `ENV_VAR_SECRET` and `CONFIG_SECRET` detection. ## Minimum Value Length Short values like `"yes"` or `"0"` are skipped by default. Adjust the threshold if needed: ```typescript theme={null} const anonymizer = createAnonymizer({ secrets: { enabled: true, minValueLength: 8, // default is 4 }, }); ``` ## Full Configuration Reference ```typescript theme={null} interface SecretsConfig { enabled: boolean; // Enable secrets detection envFiles?: string[]; // .env file paths to parse redactValues?: string[]; // Explicit values to always redact secretKeyPatterns?: RegExp[]; // Additional key name patterns minValueLength?: number; // Minimum value length (default: 4) } ``` ## Next Steps See all supported PII categories. Add your own domain-specific patterns. # Semantic Enrichment Source: https://docs.rehydra.ai/guides/semantic-enrichment Add gender and location attributes for better machine translation Semantic enrichment adds contextual attributes to PII placeholders, helping translation systems maintain grammatical correctness across languages. ## Why Semantic Attributes? Many languages have grammatical gender agreement. Without knowing the gender of a person, translation quality suffers: ``` // Without semantic enrichment Input: "Thank for the help." German: "Danke für die Hilfe." // With semantic enrichment Input: "Thank for the help." German: "Danke ihr für die Hilfe." // Correct feminine pronoun ``` ## Enable Semantic Enrichment ```typescript theme={null} import { createAnonymizer } from 'rehydra'; const anonymizer = createAnonymizer({ ner: { mode: 'quantized' }, semantic: { enabled: true, onStatus: (status) => console.log(status), } }); await anonymizer.initialize(); const result = await anonymizer.anonymize('Hello Maria Schmidt from Berlin!'); console.log(result.anonymizedText); // "Hello from !" ``` ## Semantic Attributes ### Person Gender | Attribute Value | Meaning | Example Names | | ------------------ | --------------------- | ------------------- | | `gender="male"` | Masculine name | John, Michael, Hans | | `gender="female"` | Feminine name | Maria, Sarah, Anna | | `gender="neutral"` | Ambiguous/unknown | Alex, Jordan, Sam | | `gender="unknown"` | Not found in database | — | ### Location Scope | Attribute Value | Meaning | Examples | | ----------------- | --------------------- | ----------------------------- | | `scope="city"` | City/town | Berlin, Paris, Tokyo | | `scope="country"` | Country | Germany, France, Japan | | `scope="region"` | Region/state | Bavaria, California, Hokkaido | | `scope="unknown"` | Not found in database | — | ## Semantic Data Semantic enrichment uses lookup databases (\~4 MB total): * **Name database**: 40K+ first names with gender associations (from gender-guesser) * **City database**: 25K+ cities from GeoNames (population > 15K) * **Country database**: Country names and ISO codes * **Region database**: First-level administrative divisions ### First-Use Download ```typescript theme={null} const anonymizer = createAnonymizer({ semantic: { enabled: true, autoDownload: true, // Default: auto-download if not cached onDownloadProgress: (progress) => { console.log(`${progress.file}: ${progress.percent}%`); } } }); ``` ### Manual Data Management ```typescript theme={null} import { isSemanticDataDownloaded, downloadSemanticData, clearSemanticDataCache } from 'rehydra'; // Check if data is cached const hasData = await isSemanticDataDownloaded(); // Pre-download await downloadSemanticData((progress) => { console.log(`${progress.file}: ${progress.percent}%`); }); // Clear cache await clearSemanticDataCache(); ``` ## Title Extraction When semantic enrichment is enabled, honorific titles are extracted and kept visible: ```typescript theme={null} const result = await anonymizer.anonymize('Contact Dr. Maria Schmidt'); // "Contact Dr. " ``` Supported titles: * Academic: Dr., Prof., PhD * Honorific: Mr., Mrs., Ms., Miss * Professional: Rev., Hon. * German: Herr, Frau, Dr. * French: M., Mme., Mlle. * And many more... Titles remain visible because they're often important for translation and don't reveal the person's identity on their own. ## Excluding Location Scopes When semantic enrichment is enabled, you can selectively skip anonymization of locations by their scope. This is useful when you want to keep country and region names visible while still anonymizing cities and addresses. ```typescript theme={null} const anonymizer = createAnonymizer({ ner: { mode: 'quantized' }, semantic: { enabled: true }, defaultPolicy: { excludeLocationScopes: new Set(['country', 'region']), }, }); await anonymizer.initialize(); const result = await anonymizer.anonymize( 'John Smith lives in Berlin, Germany.' ); console.log(result.anonymizedText); // "John Smith lives in , Germany." // Berlin (city) is anonymized, Germany (country) is kept visible ``` You can also apply this as a per-call override: ```typescript theme={null} await anonymizer.anonymize(text, undefined, { excludeLocationScopes: new Set(['country', 'region']), }); ``` Available scopes to exclude: | Scope | What it covers | Examples | | --------- | ----------------------------------- | ---------------------- | | `city` | Cities and towns | Berlin, Paris, Tokyo | | `country` | Countries | Germany, France, Japan | | `region` | States, provinces, regions | Bavaria, California | | `unknown` | Locations not found in the database | — | This option requires `semantic: { enabled: true }` because scope classification depends on the GeoNames lookup database. Without semantic enrichment, `excludeLocationScopes` is silently ignored. ## Locale Hints Improve detection accuracy with locale hints: ```typescript theme={null} const result = await anonymizer.anonymize( 'Bonjour Jean-Pierre de Lyon!', 'fr-FR' // French locale ); ``` The locale helps with: * Name gender inference (culture-specific names) * Title recognition (Mr. vs Herr vs M.) ## Configuration Options ```typescript theme={null} const anonymizer = createAnonymizer({ semantic: { enabled: true, autoDownload: true, onStatus: (status) => console.log(status), onDownloadProgress: (progress) => { console.log(`${progress.file}: ${progress.percent}%`); } }, defaultPolicy: { enableSemanticMasking: true, // Enable in policy (auto-set when semantic.enabled) } }); ``` ## Cache Locations Semantic data is cached locally: ### Node.js | Platform | Location | | -------- | ----------------------------------------- | | macOS | `~/Library/Caches/rehydra/semantic-data/` | | Linux | `~/.cache/rehydra/semantic-data/` | | Windows | `%LOCALAPPDATA%/rehydra/semantic-data/` | ### Browser Uses IndexedDB for cross-session persistence. ## Use Cases German, French, Spanish, and many other languages have grammatical gender. Semantic attributes help MT systems: ``` EN: "Please contact for assistance." DE: "Bitte kontaktieren Sie sie für Unterstützung." ``` Different location types use different prepositions: ``` "I'm in Berlin" (city) → "Ich bin in Berlin" "I'm in Germany" (country) → "Ich bin in Deutschland" ``` The `scope` attribute helps translation systems choose correctly. Beyond translation, semantic attributes enable: * Gender-aware text generation * Location-based content filtering * Name normalization ## Next Steps Using semantic enrichment in browsers Persist enriched PII maps # Sessions & Storage Source: https://docs.rehydra.ai/guides/sessions-storage Persist encrypted PII maps for multi-turn conversations For applications that need to persist PII maps across requests or page reloads (like chat applications), Rehydra provides session-based storage. ## Storage Providers | Provider | Environment | Persistence | Use Case | | ----------------------------- | ------------ | --------------- | ------------------------ | | `InMemoryPIIStorageProvider` | All | None | Development, testing | | `SQLitePIIStorageProvider` | Node.js, Bun | File-based | Server-side applications | | `IndexedDBPIIStorageProvider` | Browser | Browser storage | Client-side applications | `SQLitePIIStorageProvider` is **not available in browser builds**. Use `IndexedDBPIIStorageProvider` for browser applications. ## Important: Storage Requires Sessions The storage provider is **only used when you call `anonymizer.session()`**. Direct `anonymize()` calls return the PII map but don't save it: ```typescript theme={null} // ❌ Storage NOT used - you must handle the PII map yourself const result = await anonymizer.anonymize('Hello John!'); // result.piiMap is returned but NOT saved to storage // ✅ Storage IS used - auto-saves and auto-loads const session = anonymizer.session('conversation-123'); const result = await session.anonymize('Hello John!'); // result.piiMap is automatically saved to storage ``` ## Session Example ```typescript theme={null} import { createAnonymizer, InMemoryKeyProvider, SQLitePIIStorageProvider, // or IndexedDBPIIStorageProvider for browser } from 'rehydra'; // 1. Setup storage (once at app start) const storage = new SQLitePIIStorageProvider('./pii-maps.db'); await storage.initialize(); // 2. Create anonymizer with storage and key provider const anonymizer = createAnonymizer({ ner: { mode: 'quantized' }, keyProvider: new InMemoryKeyProvider(), piiStorageProvider: storage, }); await anonymizer.initialize(); // 3. Create a session for each conversation const session = anonymizer.session('conversation-123'); // 4. Anonymize - auto-saves to storage const result = await session.anonymize('Hello John Smith from Acme Corp!'); console.log(result.anonymizedText); // "Hello from !" // 5. Later (even after app restart): rehydrate - auto-loads and decrypts const translated = await translateAPI(result.anonymizedText); const original = await session.rehydrate(translated); console.log(original); // "Hello John Smith from Acme Corp!" ``` ## Multiple Conversations Each session ID maps to a separate stored PII map: ```typescript theme={null} // Different chat sessions const aliceChat = anonymizer.session('user-alice-chat'); const bobChat = anonymizer.session('user-bob-chat'); await aliceChat.anonymize('Alice: Contact me at alice@example.com'); await bobChat.anonymize('Bob: My number is +49 30 123456'); // Each session has independent storage await aliceChat.rehydrate(translatedText1); // Uses Alice's PII map await bobChat.rehydrate(translatedText2); // Uses Bob's PII map ``` ## Multi-Message Conversations Within a session, entity IDs are consistent across multiple `anonymize()` calls: ```typescript theme={null} const session = anonymizer.session('chat-123'); // Message 1: User provides contact info const msg1 = await session.anonymize('Contact me at user@example.com'); // → "Contact me at " // Message 2: References same email + new one const msg2 = await session.anonymize('CC: user@example.com and admin@example.com'); // → "CC: and " // ↑ Same ID (reused) ↑ New ID // All messages can be rehydrated correctly await session.rehydrate(msg1.anonymizedText); // ✓ await session.rehydrate(msg2.anonymizedText); // ✓ ``` ## Session Interface ```typescript theme={null} interface AnonymizerSession { readonly sessionId: string; // Anonymize text and save to storage anonymize( text: string, locale?: string, policy?: Partial ): Promise; // Load PII map and restore original values rehydrate(text: string): Promise; // Load stored PII map load(): Promise; // Delete session from storage delete(): Promise; // Check if session exists exists(): Promise; } ``` ## SQLite Provider (Node.js / Bun) ```typescript theme={null} import { SQLitePIIStorageProvider } from 'rehydra'; // File-based database const storage = new SQLitePIIStorageProvider('./data/pii-maps.db'); await storage.initialize(); // Or in-memory for testing const testStorage = new SQLitePIIStorageProvider(':memory:'); await testStorage.initialize(); ``` **Dependencies:** * **Bun**: Uses built-in `bun:sqlite` (no additional install) * **Node.js**: Requires `better-sqlite3`: ```bash theme={null} npm install better-sqlite3 ``` ## IndexedDB Provider (Browser) ```typescript theme={null} import { createAnonymizer, InMemoryKeyProvider, IndexedDBPIIStorageProvider, } from 'rehydra'; // Custom database name (defaults to 'rehydra-pii-storage') const storage = new IndexedDBPIIStorageProvider('my-app-pii'); const anonymizer = createAnonymizer({ ner: { mode: 'quantized' }, keyProvider: new InMemoryKeyProvider(), piiStorageProvider: storage, }); await anonymizer.initialize(); // Use sessions as usual const session = anonymizer.session('browser-chat-123'); ``` ## Data Retention Entries persist forever by default. Manage data with: ```typescript theme={null} // Delete entries older than 7 days const sevenDaysAgo = new Date(Date.now() - 7 * 24 * 60 * 60 * 1000); const deletedCount = await storage.cleanup(sevenDaysAgo); // Delete specific session await session.delete(); // List all stored sessions const sessionIds = await storage.list(); ``` ## Storage Provider Interface Implement custom storage: ```typescript theme={null} interface PIIStorageProvider { initialize(): Promise; save(sessionId: string, data: StoredPIIMap): Promise; load(sessionId: string): Promise; delete(sessionId: string): Promise; exists(sessionId: string): Promise; list(options?: ListOptions): Promise; cleanup(olderThan: Date): Promise; close(): Promise; } ``` ## Without Storage (Simple One-Off Usage) For simple use cases without persistence: ```typescript theme={null} import { createAnonymizer, decryptPIIMap, rehydrate, InMemoryKeyProvider } from 'rehydra'; const keyProvider = new InMemoryKeyProvider(); const anonymizer = createAnonymizer({ ner: { mode: 'quantized' }, keyProvider, }); await anonymizer.initialize(); // Anonymize const result = await anonymizer.anonymize('Hello John Smith!'); // Translate const translated = await translateAPI(result.anonymizedText); // Rehydrate manually const key = await keyProvider.getKey(); const piiMap = await decryptPIIMap(result.piiMap, key); const original = rehydrate(translated, piiMap); ``` ## Next Steps Storage considerations for browsers Understand PII map encryption # Streaming Anonymization Source: https://docs.rehydra.ai/guides/streaming Anonymize text in real-time as it arrives chunk by chunk The streaming API processes text incrementally, making it ideal for real-time LLM output, file processing, and other scenarios where text arrives in chunks. Streaming is only available in Node.js and Bun. It is not included in the browser build. ## Basic Usage ```typescript theme={null} import { createAnonymizerStream } from 'rehydra/streaming'; const stream = await createAnonymizerStream({ anonymizer: { ner: { mode: 'quantized' }, }, }); // Pipe through the stream process.stdin.pipe(stream).pipe(process.stdout); ``` ## How It Works The streaming anonymizer uses a **sentence buffer** that accumulates text until a sentence boundary is detected, then anonymizes complete sentences. This ensures NER has enough context for accurate detection. ``` Incoming chunks → Sentence Buffer → Anonymize → Output (accumulates) (per sentence) ``` An overlap region preserves context between flushes so that entities spanning buffer boundaries are detected correctly. ## Stream Configuration ```typescript theme={null} import { createAnonymizerStream } from 'rehydra/streaming'; const stream = await createAnonymizerStream({ // Anonymizer configuration (same as createAnonymizer) anonymizer: { ner: { mode: 'quantized' }, semantic: { enabled: true }, }, // Policy override policy: { enableLeakScan: true }, // Locale hint locale: 'de-DE', // Buffer configuration buffer: { overlapChars: 100, // Context overlap between flushes (default: 100) maxBufferSize: 8192, // Force flush at this size (default: 8192) minBufferSize: 50, // Minimum before flushing (default: 50) sentenceBoundary: /[.!?]\s+/, // Custom boundary regex }, // Session persistence sessionId: 'chat-123', keyProvider: myKeyProvider, piiStorageProvider: myStorage, saveIntervalMs: 5000, // Periodic PII map saves (ms) // Events onChunk: (event) => { console.log(`Chunk: ${event.anonymizedText}`); console.log(`Entities so far: ${event.totalEntities}`); }, onFinish: (event) => { console.log(`Done. Total entities: ${event.totalEntities}`); console.log(`Total time: ${event.totalProcessingTimeMs}ms`); }, }); ``` ## Low-Latency Mode For real-time LLM token streams where latency matters more than NER accuracy: ```typescript theme={null} const stream = await createAnonymizerStream({ buffer: { lowLatency: true, // Disables NER, reduces buffer sizes, flushes aggressively }, }); ``` Low-latency mode: * Disables NER (regex-only detection) * Reduces buffer sizes for faster flushing * Optimized for token-by-token LLM output ## Events ### onChunk Fires after each buffered chunk is anonymized: ```typescript theme={null} interface StreamChunkEvent { anonymizedText: string; entities: DetectedEntity[]; totalEntities: number; processingTimeMs: number; } ``` ### onFinish Fires when the stream ends: ```typescript theme={null} interface StreamFinishEvent { totalEntities: number; piiMap?: EncryptedPIIMap; totalProcessingTimeMs: number; } ``` ## Accessing the PII Map After the stream ends, retrieve the cumulative PII map: ```typescript theme={null} stream.on('finish', () => { const piiMap = stream.getPiiMap(); const stats = stream.stats; }); ``` ## Session Integration Persist PII maps across streaming sessions: ```typescript theme={null} import { createAnonymizerStream } from 'rehydra/streaming'; import { InMemoryKeyProvider, SQLitePIIStorageProvider } from 'rehydra'; const storage = new SQLitePIIStorageProvider('./pii.db'); await storage.initialize(); const stream = await createAnonymizerStream({ sessionId: 'chat-123', keyProvider: new InMemoryKeyProvider(), piiStorageProvider: storage, saveIntervalMs: 5000, // Save PII map every 5 seconds }); ``` If a session already has a stored PII map, the stream loads it and continues with consistent entity IDs. ## Next Steps Automatic anonymization for LLM API calls Complete streaming API documentation # Introduction Source: https://docs.rehydra.ai/index Privacy tools for protecting sensitive data in code, terminals, and AI workflows Rehydra is a privacy layer for working with sensitive text and secrets. It helps you detect protected values locally, replace them with safe placeholders, and restore them when you need the originals again. Build privacy-preserving applications in Node.js, Bun, or the browser. Anonymize, inspect, and rehydrate text directly from the terminal. Prevent coding agents from leaking PII and secrets to LLMs. ## Three Ways to Use Rehydra ### SDK Use the `rehydra` package when you want privacy controls inside your application. * Detect structured PII such as emails, phone numbers, credit cards, IPs, and URLs * Add optional NER-based detection for names, organizations, and locations * Encrypt original values for later rehydration * Integrate with apps, services, agents, proxies, and streaming workflows ### CLI Tool Use `@rehydra/cli` when you want to work with files or piped text without writing code. * Anonymize text from files or stdin * Inspect what would be detected before changing anything * Rehydrate text using a saved encrypted PII map * Fit Rehydra into shell scripts and local workflows ### OpenCode Plugin Use `@rehydra/opencode` when you want OpenCode to automatically scrub PII and secrets before they are sent to an LLM. * Auto-detect PII (emails, phones, credit cards) and secrets (API keys, JWTs, connection strings) in all messages * Optionally scan `.env` files to catch exact secret values anywhere in text * Restore original values before local tool execution * Add privacy protection to agent workflows with minimal setup ## How Rehydra Works Rehydra identifies sensitive values on your machine before they leave your environment. It swaps originals for stable placeholders that downstream systems can handle safely. Rehydra can decrypt and rehydrate the original values in the right local context. ## What You Can Protect Emails, phone numbers, IBANs, credit cards, IP addresses, URLs, and more. Names, organizations, and locations with the optional NER model. API keys, tokens, connection strings, and other environment-based credentials. Original values are stored in encrypted form for later restoration. ## Start Here Start with the SDK, CLI, or OpenCode plugin. See install commands for each environment and package. Learn the terminal workflow and available commands. Set up automatic PII and secret scrubbing for OpenCode. # Installation Source: https://docs.rehydra.ai/installation Install Rehydra for Node.js, Bun, or browsers ## CLI For terminal usage without writing code: ```bash npm theme={null} npm install -g @rehydra/cli ``` ```bash bun theme={null} bun add -g @rehydra/cli ``` ```bash pnpm theme={null} pnpm add -g @rehydra/cli ``` See the [CLI Guide](/guides/cli) for commands and options. ## Node.js Install the SDK from npm: ```bash theme={null} npm install rehydra ``` The NER model requires `onnxruntime-node`, which is included as an optional dependency and will be installed automatically on supported platforms. ### SQLite Storage (Optional) For persistent PII map storage with SQLite: ```bash theme={null} npm install better-sqlite3 ``` ## Bun Bun requires `onnxruntime-web` since `onnxruntime-node` is a native Node.js addon: ```bash theme={null} bun add rehydra onnxruntime-web ``` Usage is identical to Node.js — the library auto-detects the runtime. ## Browser (with Bundler) When using Vite, webpack, esbuild, or other modern bundlers: ```bash theme={null} npm install rehydra onnxruntime-web ``` The browser-safe entry point is automatically selected via [conditional exports](https://nodejs.org/api/packages.html#conditional-exports). This entry point excludes Node.js-specific modules like SQLite storage. ### Vite Configuration No special configuration needed. Vite automatically resolves the browser build. ### Webpack Configuration No special configuration needed. Webpack 5+ resolves conditional exports correctly. ## Browser (without Bundler) For direct browser usage without a bundler: ```html theme={null} ``` ## Conditional Exports The package uses conditional exports to provide the right build for each environment: ```json theme={null} { "exports": { ".": { "browser": "./dist/browser.js", "node": "./dist/index.js", "default": "./dist/index.js" } } } ``` ### Explicit Imports If needed, you can explicitly import a specific build: ```typescript theme={null} // Browser-only build (excludes SQLite, Node.js fs, etc.) import { createAnonymizer } from 'rehydra/browser'; // Node.js build (includes everything) import { createAnonymizer, SQLitePIIStorageProvider } from 'rehydra/node'; // SQLite storage only (Node.js only) import { SQLitePIIStorageProvider } from 'rehydra/storage/sqlite'; // Streaming anonymization (Node.js/Bun only) import { createAnonymizerStream } from 'rehydra/streaming'; // LLM proxy middleware (Node.js/Bun only) import { wrapLLMClient, createRehydraProxy } from 'rehydra/proxy'; ``` ## What's Included ### Browser Build Includes * All recognizers (email, phone, IBAN, credit card, etc.) * NER model support (with `onnxruntime-web`) * Semantic enrichment * `InMemoryPIIStorageProvider` * `IndexedDBPIIStorageProvider` * All crypto utilities ### Browser Build Excludes * `SQLitePIIStorageProvider` (use `IndexedDBPIIStorageProvider` instead) * Node.js `fs`, `path`, `os` modules ## Requirements | Environment | Version | Notes | | ----------- | ----------------------------------------------- | ------------------------------ | | Node.js | ≥ 18.0.0 | Uses native `onnxruntime-node` | | Bun | ≥ 1.0.0 | Requires `onnxruntime-web` | | Browsers | Chrome 86+, Firefox 89+, Safari 15.4+, Edge 86+ | Uses OPFS for model storage | ## Verifying Installation ```typescript theme={null} import { anonymizeRegexOnly } from 'rehydra'; const result = await anonymizeRegexOnly('Contact: test@example.com'); console.log(result.anonymizedText); // "Contact: " ``` If you see the anonymized output, installation is complete! ## Next Steps Learn the basics of using Rehydra Set up the NER model for detecting names and locations # Quick Start Source: https://docs.rehydra.ai/quickstart Choose the right Rehydra product for your workflow Rehydra ships as three product surfaces: the SDK for application code, the CLI for terminal workflows, and the OpenCode plugin for agent privacy. Start with the one that matches how you work. ## Choose Your Starting Point Add Rehydra to your app and control anonymization in code. Protect files and piped text from the terminal. Scrub secrets before OpenCode sends them to an LLM. ## SDK Use the SDK when you are building a product or integrating privacy into an existing system. ```bash npm theme={null} npm install rehydra ``` ```bash bun theme={null} bun add rehydra onnxruntime-web ``` ```bash pnpm theme={null} pnpm add rehydra ``` ```typescript theme={null} import { anonymizeRegexOnly } from 'rehydra'; const result = await anonymizeRegexOnly('Contact john@example.com'); console.log(result.anonymizedText); // "Contact " ``` Best for: * Server-side apps and backend services * Browser-based privacy workflows * Agent, proxy, or streaming integrations * Teams that need fine-grained control over detection and rehydration ## CLI Tool Use the CLI when you want to protect text quickly without writing application code. ```bash npm theme={null} npm install -g @rehydra/cli ``` ```bash bun theme={null} bun add -g @rehydra/cli ``` ```bash pnpm theme={null} pnpm add -g @rehydra/cli ``` ```bash theme={null} rehydra anonymize input.txt -o output.txt ``` Best for: * Local document processing * Shell scripts and automation * Inspecting sensitive text before sharing it * Teams that want a no-code entry point ## OpenCode Plugin Use the OpenCode plugin when your coding agent should never send raw secrets to an LLM provider. ```bash npm theme={null} npm install @rehydra/opencode ``` ```bash bun theme={null} bun add @rehydra/opencode ``` ```bash pnpm theme={null} pnpm add @rehydra/opencode ``` ```json theme={null} { "plugin": ["@rehydra/opencode"] } ``` Best for: * OpenCode-based agent workflows * Repositories with sensitive `.env` files * Local development environments using AI assistants * Teams adding privacy guardrails to coding agents ## Shared Model All three product surfaces follow the same core pattern: Sensitive values are identified locally. Originals are replaced with safe placeholders. Original values can be rehydrated locally when needed. ## Next Steps View environment-specific install details for the SDK. Explore commands, options, and examples. Configure secret scrubbing for OpenCode. Dive into the SDK APIs and advanced features.