Visitar URL original
[Bug] Search panics on disk-write failure and crashes the entire all-in-one process · Issue #3796 · opencloud-eu/opencloud · GitHub
Skip to content

[Bug] Search panics on disk-write failure and crashes the entire all-in-one process #3796

Open

Description

@suse-coder

Describe the bug

When a Bleve/zapx segment persist fails (disk quota exceeded), search panics with assignment to entry in nil map in invertedIndexCache. On all-in-one that kills the whole process (search, proxy, storage-users, NATS, postprocessing), exit 2, CrashLoopBackOff.

Steps to reproduce

  1. Run all-in-one OpenCloud with Bleve v5 / zapx v17.
  2. Hit a write failure on bleve-v5/store/*.zap (disk quota exceeded) during an index batch.
  3. Have a search query in flight at the same time.

Expected behavior

A failed persist is an error in search. The rest of the process keeps running.

Actual behavior

{"level":"error","service":"search","error":"segment: /var/lib/opencloud/search/bleve-v5/store/000000003df6.zap persist err: write /var/lib/opencloud/search/bleve-v5/store/000000003df6.zap: disk quota exceeded","time":"2026-10-22T13:39:47Z","line":"github.com/opencloud-eu/opencloud/services/search/pkg/search/service.go:690","message":"error adding updating the resource in the index"}

Then:

panic: assignment to entry in nil map

github.com/blevesearch/zapx/v17.(*invertedIndexCache).insertLOCKED(...)
        github.com/blevesearch/zapx/v17@v17.2.3/inverted_text_cache.go:94

Two goroutines hit the same nil map — search query (SearchInContext → TermFieldReader) and index write (Scorch.Batch → DocNumbers). Proxy had been serving (GET /graph/v1.0/me → 200); Kubernetes then saw :9200 connection refused. df still showed free space; the write error was disk quota exceeded.

Setup

All-in-one Helm, 1 replica, Recreate. Data PVC CephFS RWX 300 Gi at /var/lib/opencloud.

Details

Image=docker.io/opencloudeu/opencloud-rolling:8.1.0
STORAGE_USERS_DRIVER=decomposed
STORAGE_SYSTEM_DRIVER=decomposed
SEARCH_EXTRACTOR_TYPE=tika
OC_ASYNC_UPLOADS=true

Additional context

bleve/v2@v2.6.1, zapx/v17@v17.2.3, reva/v2@v2.51.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions