Configuration
The Python binding shares the same Rust core configuration. Settings can be provided through the Config builder, environment variables, or a properties file. When the same parameter appears in multiple sources, the highest-priority source wins:
Priority (highest → lowest):
1. Environment variables (GOOSEFS_*)
2. Properties config file (goosefs-site.properties)
3. Built-in defaults
Minimal Setup
from goosefs import Config
# Single master (GOOSEFS_* env overrides are applied automatically)
cfg = Config("127.0.0.1:9200")
# From a goosefs-site.properties file
cfg = Config.from_properties_file("/etc/goosefs/goosefs-site.properties")
# From a gfs:// URI (optional root path; auth via env / properties)
cfg = Config.from_uri("gfs://127.0.0.1:9200/data")
Environment Variables
| Variable | Purpose |
|---|---|
GOOSEFS_MASTER_ADDR | Master host:port (or comma-separated HA list) |
GOOSEFS_AUTH_TYPE | nosasl / simple / custom |
GOOSEFS_AUTH_USERNAME | Username for SIMPLE auth |
GOOSEFS_USER_FILE_REPLICATION_NUMBER | Block-worker selection count (default 1) |
GOOSEFS_USER_FILE_READ_MAX_NODE_RETRY | Read candidate pool / Java maxRetryNode (default 3) |
GOOSEFS_USER_FILE_CHECK_BLOCK_REPLICAS | CheckBlocks probe count; 0 disables (default) |
GOOSEFS_MASTER_CONNECTION_POOL_SIZE | Master gRPC channel pool size (default 1) |
GOOSEFS_MASTER_POOL_SCHEDULE | roundrobin / p2c |
GOOSEFS_WORKER_CONNECTION_POOL_SIZE | Per-worker gRPC channel pool size |
GOOSEFS_USER_NETWORK_RPC_CONNECT_TIMEOUT | gRPC connect timeout (5sec / 5000ms / 5000) |
GOOSEFS_USER_NETWORK_RPC_TIMEOUT | Per-RPC request timeout (same time format) |
GOOSEFS_USER_NETWORK_VPC_MAPPING_ENABLED | Use VPC mapping addresses (true/false) |
GOOSEFS_METADATA_CACHE_ENABLED | Client metadata cache switch (default true) |
GOOSEFS_METADATA_CACHE_EXPIRATION | Metadata cache TTL (10min, 30s, raw ms) |
GOOSEFS_METADATA_CACHE_MAX_SIZE | Metadata cache LRU capacity (default 100000) |
GOOSEFS_FILE_METADATA_SYNC_INTERVAL | Metadata sync interval (parseTimeSize; default -1. 0 skips cache on every get/list) |
GOOSEFS_FILE_METADATA_LOAD_TYPE | ONCE / ALWAYS / NEVER (default ONCE. ALWAYS skips listing cache) |
GOOSEFS_USER_FILE_PERSIST_ON_RENAME | Async-persist destination on rename (default false) |
Write / Read Types
| Enum | Typical use |
|---|---|
WriteType.MustCache | Cache only (no UFS persist) |
WriteType.TryCache | Try cache first, fall back to Through on error |
WriteType.CacheThrough | Write cache + UFS synchronously |
WriteType.Through | Write UFS directly |
WriteType.AsyncThrough | Write cache, persist UFS asynchronously |
from goosefs import Config, Goosefs, WriteType
cfg = Config("127.0.0.1:9200")
fs = Goosefs(cfg) # sync; use AsyncGoosefs for async
fs.write_file("/data/file.bin", b"payload", write_type=WriteType.CacheThrough)
Master Connection Pool
The master connection pool spreads concurrent metadata RPCs across multiple HTTP/2 channels. Default size is 1 (single channel, backward-compatible) with round-robin scheduling. Raise to 4-8 with P2C scheduling for high-concurrency remote scenarios.
# Via env
# export GOOSEFS_MASTER_CONNECTION_POOL_SIZE=8
# export GOOSEFS_MASTER_POOL_SCHEDULE=p2c
# Via properties file
# goosefs.user.master.connection.pool.size=8
# goosefs.user.master.pool.schedule=p2c
# Via storage options (OpenDAL / Lance)
# storage_options={"goosefs_master_connection_pool_size": "8", ...}
Connection Timeouts and VPC Mapping
These fields are readable on Config (connect_timeout_ms, request_timeout_ms, use_vpc_mapping) and can be set via properties, env, or a properties file:
| Property key | Env var | Default |
|---|---|---|
goosefs.user.network.rpc.connect.timeout | GOOSEFS_USER_NETWORK_RPC_CONNECT_TIMEOUT | 30s |
goosefs.user.network.rpc.timeout | GOOSEFS_USER_NETWORK_RPC_TIMEOUT | 5min |
goosefs.user.network.vpc.mapping.enabled | GOOSEFS_USER_NETWORK_VPC_MAPPING_ENABLED | false |
Timeouts accept Java parseTimeSize (5sec, 5000ms, 5000).
cfg = Config("127.0.0.1:9200", properties={
"goosefs.user.network.rpc.connect.timeout": "5sec",
"goosefs.user.network.rpc.timeout": "7000",
"goosefs.user.network.vpc.mapping.enabled": "true",
})
assert cfg.connect_timeout_ms == 5000
assert cfg.request_timeout_ms == 7000
assert cfg.use_vpc_mapping is True
Client Local Page Cache (opt-in)
Disabled by default. Enable via env or properties:
| Property key | Env var | Default |
|---|---|---|
goosefs.user.client.cache.enabled | GOOSEFS_USER_CLIENT_CACHE_ENABLED | false |
goosefs.user.client.cache.page.size | GOOSEFS_USER_CLIENT_CACHE_PAGE_SIZE | 1048576 (1 MB) |
goosefs.user.client.cache.size | GOOSEFS_USER_CLIENT_CACHE_SIZE | 21474836480 (20 GiB) |
goosefs.user.client.cache.dirs | GOOSEFS_USER_CLIENT_CACHE_DIRS | /tmp/goosefs_cache |
goosefs.user.client.cache.sync.read.enabled | GOOSEFS_USER_CLIENT_CACHE_SYNC_READ_ENABLED | false (Linux only; analytical workloads on local NVMe — see Page Cache → Sync pread read mode) |
See Page Cache for a full walkthrough.
Client Metadata Cache (on by default)
Enabled by default (the Java client defaults it to false). get_status / exists / open_file / non-recursive list_status share one process-local TTL-bounded LRU (status + listing + negative cache); mkdir / delete / rename invalidate the path and its parent.
| Property key | Env var | Default |
|---|---|---|
goosefs.user.metadata.cache.enabled | GOOSEFS_METADATA_CACHE_ENABLED | true |
goosefs.user.metadata.cache.max.size | GOOSEFS_METADATA_CACHE_MAX_SIZE | 100000 |
goosefs.user.metadata.cache.expiration.time | GOOSEFS_METADATA_CACHE_EXPIRATION | 10min |
goosefs.user.file.metadata.sync.interval | GOOSEFS_FILE_METADATA_SYNC_INTERVAL | -1 |
goosefs.user.file.metadata.load.type | GOOSEFS_FILE_METADATA_LOAD_TYPE | ONCE |
This replaces the removed GOOSEFS_FILE_INFO_CACHE_TTL_MS / GOOSEFS_FILE_INFO_CACHE_CAPACITY knobs. See Metadata Cache for semantics, env vars, and metrics.
Full Parameter Reference
The complete field / env / properties / storage-options matrix lives in the repository: