Move the workspace sync engine into @deepnote/cloud-sync #563

Objective

deepnote sync (packages/cli/src/commands/sync.ts) can only be used as a CLI command. It resolves its root from process.cwd(), loads .env into process.env, prompts through @inquirer/prompts and prints through the CLI output module.

After this change, @deepnote/cloud-sync exports syncWorkspace(options), which syncs every project of a Deepnote workspace with a local folder. The caller passes the root, token and API URL explicitly, supplies a policy that decides how each conflict is resolved, and receives progress through an event callback. deepnote sync becomes a thin adapter over it, with unchanged flags, output, JSON and exit codes. @deepnote/cli is the first consumer; the MCP server (#544) and the VS Code extension follow.

Related: #543, #544.

Scope

• In scope:
  • Moving the engine and its helpers from commands/sync.ts to packages/cloud-sync/src/sync-workspace.ts.
  • The library contract: options, result, conflicts, events and cancellation.
  • The CLI adapter.
  • Moving and converting the engine test suite.
  • New CLI adapter tests.
  • Package dependency and README updates.
• Out of scope:
  • New sync behavior: a per-project filter, a reusable dry-run plan, an override that pushes local changes for a project changed on both sides, AbortSignal cancellation.
  • JSON error output.
  • Changes to option registration in cli.ts.
  • Changes to @deepnote/cloud.
  • Publishing and Streamlit code.
  • Changes to docs/ or skills/.

Relevant context

Precondition. packages/cloud-sync exists and owns sync-manifest.ts, sync-paths.ts and fs-errors.ts (added in #554). commands/sync.ts imports the manifest and path symbols from @deepnote/cloud-sync, and commands/sync.test.ts wraps saveSyncManifest and assertNoSymbolicLinkAncestors through vi.mock('@deepnote/cloud-sync', …).

packages/cli/src/commands/sync.ts today

• CLI-only symbols that stay in the CLI:

  • CONFLICT_MODES and ConflictMode ('ask' | 'skip' | 'override').
  • parseSyncConcurrency, which throws commander's InvalidArgumentError.
  • SyncOptions: url, token, allFiles, onConflict, deleteMissingNotebooks, prune, dryRun, output?: 'json' and concurrency.
  • renderOutcomeLine and renderHumanSummary.
  • createSyncAction. It prints outputJson(result) or renderHumanSummary, sets process.exitCode = 1 when !success, and through program.error maps MissingTokenError to exit code 2 and every other error to exit code 1.
  • cli.ts imports CONFLICT_MODES, DEFAULT_SYNC_CONCURRENCY, parseSyncConcurrency and createSyncAction.

• Engine symbols that move:

  • DEFAULT_SYNC_CONCURRENCY (8).
  • ProjectSyncOutcome, and SyncResult ({success, root, dryRun, projects, untrackedFiles}, which is also the -o json shape).
  • SyncContext, SyncCancelledError, throwIfCancelled, emit, assertBufferedProjectFileSize.
  • canonicalProjectHash, parseDocumentLoosely, readExportModifiedAt, SyncStep, classifySyncStep, resolveConflict.
  • writeFileEnsuringDir, pathExists, toAbsolute, readLocalNotebookFiles, writeProjectNotebooks, moveTrackedProjectDir.
  • syncProjectFiles, the private PushOutcome, pushProject, listLocalFilesRecursive, describeCloudFileDivergence, PlannedFileUpload, uploadProjectFiles, syncOneProject, findUntrackedDeepnoteFiles.
  • The body of syncWorkspace(dir, options).

• Environment coupling in syncWorkspace:

  • path.resolve(process.cwd(), dir ?? '.').
  • fs.mkdir(rootDir), unless it is a dry run.
  • dotenv.config({ path: <root>/.env, quiet: true }).
  • resolveToken(options.token), which leads to MissingTokenError when there is no token.
  • canPrompt = stdin.isTTY && stdout.isTTY && !output.
  • conflictMode = requested === 'ask' && (!canPrompt || dryRun) ? 'skip' : requested. When the mode is ask and canPrompt is false, it also calls debug('No interactive terminal; conflicts will be skipped. Use --on-conflict to decide up front.').
  • Progress lines go through emit(ctx, log, …), and only when output is unset.

• resolveConflict(ctx, question, overrideLabel):

  • For skip and override it returns conflictMode immediately.
  • For ask it chains a select prompt (choices Skip this project for now and overrideLabel) onto ctx.promptQueue, so only one prompt is open at a time. While the prompt is open it sets ctx.heldOutput = [], and flushes the held output in order afterwards.
  • If the prompt rejects with ExitPromptError, it sets ctx.cancelled, and queued prompts reject with the same error.
  • syncOneProject rethrows prompt exits and SyncCancelledError instead of recording an error outcome.
  • Workers stop at their throwIfCancelled checkpoints. After Promise.allSettled, a rejected worker triggers a final manifest save (skipped in a dry run), then the rejection is rethrown.

• The five resolveConflict call sites, and what override does at each:

  Call site	Situation	What override does
  pushProject	Empty local directory with deleteMissingNotebooks	Push and delete every cloud notebook
  pushProject	Import returns 409 (other than "Project is suspended")	Re-import with force: true
  uploadProjectFiles	Working files changed in Deepnote; one call per project with {relPath, conflict} entries	Upload the local copies
  syncOneProject	Tracked project changed on both sides	Pull the cloud version
  syncOneProject	Untracked local directory that differs from the cloud	Pull the cloud version

• Output emitted by the engine:

  • Warnings via emit(ctx, warn, …): one in writeProjectNotebooks, two in syncProjectFiles and two in uploadProjectFiles.
  • Debug lines Downloaded ${project.name}: ${entry.path} (${entry.size} bytes) and Uploaded ${project.name}: ${relPath} (${bytes.length} bytes).
  • Progress: Listing projects from ${baseUrl}… (dim), and one renderOutcomeLine(outcome) per project that finishes, is pruned or is missing in the cloud.

• Engine imports that move with it:

  • node:fs/promises, node:path and the global crypto.randomUUID.
  • parseYaml from @deepnote/blocks.
  • ApiError from @deepnote/database-integrations.
  • From @deepnote/cloud: deleteProjectFile, downloadProjectFile, exportProject, getProjectDetail, importProject, listAllProjects, uploadProjectFile, MAX_BUFFERED_PROJECT_FILE_BYTES, and the types ExportedNotebookFile, ImportedNotebook, ProjectFileEntry and SyncProject.
  • isErrnoENOENT now comes from packages/cloud-sync/src/fs-errors.ts.

packages/cli/src/commands/sync.test.ts today (72 tests)

• Calls syncWorkspace(tempDir, { ...baseOptions, … }) directly.
• Fakes the API with vi.spyOn(global, 'fetch') in installCloud.
• Mocks select from @inquirer/prompts.
• Stubs TTYs with Object.defineProperty(process.stdin/stdout, 'isTTY') and withTty.
• Reads progress from console.log and warnings from console.error in nine tests, including the exact string 'Skipping file with unsafe path in "C": ../evil.csv'.
• Spies on saveSyncManifest and assertNoSymbolicLinkAncestors to observe manifest saves and the order in which projects start.
• createSyncAction, renderHumanSummary and the JSON output have no tests.

Name clash. @deepnote/local-runner already exports types named SyncOptions and SyncResult, so the library's public names must differ.

Completion criteria

• @deepnote/cloud-sync exports syncWorkspace, DEFAULT_SYNC_CONCURRENCY and the types listed in step 1.
• packages/cli/src/commands/sync.ts no longer contains engine code: no exportProject or importProject calls.
• deepnote sync produces the same stdout, stderr, -o json document and exit code as before for every flag combination, as the converted and new tests show.
• syncWorkspace never reads or writes process.env: with a .env file containing DEEPNOTE_TOKEN in rootDir, process.env is unchanged after a run.
• With a function policy, the function:
  • is never called concurrently;
  • is never called in a dry run;
  • when it rejects, makes syncWorkspace reject with that same error once every worker has settled. Cancellation takes effect at the existing throwIfCancelled checkpoints: no project, import, file download, prune deletion or file replacement starts after the rejection. Work already past its checkpoint completes, including a file replacement whose delete was already sent (its upload and any cleanup delete still run) and a notebook write batch already in progress. Unless it is a dry run, a final manifest save is attempted before the rejection; a failed save is ignored.
• syncWorkspace rejects a concurrency that is not a positive integer with RangeError before it creates rootDir or calls the API.
• Outside *.test.ts files, packages/cloud-sync/src still has no import from commander, chalk, ora, @inquirer/prompts, dotenv or @deepnote/cli, and no process. reference.

Implementation

1. Define the library contract

   • File to create: packages/cloud-sync/src/sync-workspace.ts.

   • Exported API:

     export const DEFAULT_SYNC_CONCURRENCY = 8
     export type SyncConflictDecision = 'skip' | 'override'
     export type SyncConflict =
       | { kind: 'empty-local-directory'; projectId: string; projectName: string }
       | { kind: 'cloud-changed-after-local-edit'; projectId: string; projectName: string }
       | { kind: 'working-files-changed'; projectId: string; projectName: string; projectDir: string; files: { path: string; reason: string }[] }
       | { kind: 'changed-on-both-sides'; projectId: string; projectName: string; projectDir: string }
       | { kind: 'untracked-local-directory'; projectId: string; projectName: string; projectDir: string }
     export type SyncConflictPolicy =
       | SyncConflictDecision
       | ((conflict: SyncConflict) => Promise<SyncConflictDecision>)
     export type SyncEvent =
       | { kind: 'listing-projects'; baseUrl: string }
       | { kind: 'project-outcome'; outcome: ProjectSyncOutcome }
       | { kind: 'warning'; message: string }
       | { kind: 'file-transferred'; direction: 'download' | 'upload'; projectName: string; path: string; bytes: number }
     export interface WorkspaceSyncOptions {
       rootDir: string
       baseUrl: string
       token: string
       allFiles?: boolean
       deleteMissingNotebooks?: boolean
       prune?: boolean
       dryRun?: boolean
       concurrency?: number             // positive integer; default DEFAULT_SYNC_CONCURRENCY
       onConflict?: SyncConflictPolicy  // default 'skip'
       onEvent?: (event: SyncEvent) => void
     }
     export interface ProjectSyncOutcome { /* moved unchanged from commands/sync.ts */ }
     export interface WorkspaceSyncResult { /* moved unchanged from SyncResult in commands/sync.ts */ }
     export function syncWorkspace(options: WorkspaceSyncOptions): Promise<WorkspaceSyncResult>

   • Each SyncConflict variant has TSDoc stating what override does (see the table in Relevant context). The TSDoc on projectDir states that it is a POSIX path relative to rootDir, e.g. Analytics/Sales report. The TSDoc on files[].path and on the file-transferred event's path states that each is the working file's project-relative POSIX path: its path in Deepnote, and locally relative to <rootDir>/<projectDir>/.files, never prefixed with projectDir, e.g. _deepnote_static/index.html.

   • Export everything above from packages/cloud-sync/src/index.ts.

2. Move the engine behind the contract
   • Move the engine symbols listed in Relevant context from commands/sync.ts into sync-workspace.ts. They stay unchanged except as follows:
     • syncWorkspace first validates options.concurrency: when it is defined and !Number.isInteger(concurrency) || concurrency < 1, the call rejects with new RangeError('concurrency must be a positive integer') before it creates rootDir or calls the API, following projectFileTransferLimit in packages/cloud/src/sync.ts. The CLI keeps parseSyncConcurrency and its InvalidArgumentError message.
     • SyncContext holds WorkspaceSyncOptions and the onConflict policy instead of the CLI's SyncOptions and conflictMode.
     • syncWorkspace resolves rootDir with path.resolve(options.rootDir), creates it unless dryRun is set, and uses options.baseUrl and options.token as given. It does not call dotenv, resolveToken or process.cwd(), and it does no TTY checks.
     • emit delivers SyncEvents to options.onEvent and keeps the hold-and-flush behavior:
       • The warning call sites emit { kind: 'warning', message } with the current message text.
       • The two debug call sites emit file-transferred, using entry.size for downloads and bytes.length for uploads.
       • The Listing projects progress line becomes a listing-projects event.
       • Every progress(renderOutcomeLine(outcome)) call becomes a project-outcome event.
     • resolveConflict(ctx, conflict) takes a SyncConflict:
       • A string policy returns immediately, with no queueing and no holding.
       • A function policy is treated as 'skip' when dryRun is set. Otherwise the function is called through the existing promise queue, so only one call is outstanding at a time. Events are held while a call is pending and flushed in order once it settles.
       • If the function rejects, the engine records that error as the cancellation reason and sets ctx.cancelled. Queued calls then reject with the same error without calling the function.
     • syncOneProject rethrows the cancellation-reason error and SyncCancelledError, replacing the isPromptExit check. Delete isPromptExit. Worker shutdown, the final manifest save on rejection, and the rethrow stay as they are. Keep every throwIfCancelled checkpoint where it is; add none between deleteProjectFile and uploadProjectFile in uploadProjectFiles.
     • Each call site builds its SyncConflict variant from data it already has: project.id, project.name, plan.projectDir, and the {relPath, conflict} list mapped to {path, reason}.
   • In packages/cloud-sync/package.json, add "@deepnote/blocks": "workspace:*" and "@deepnote/database-integrations": "workspace:*" to dependencies if they are not there yet. Add both to external in packages/cloud-sync/tsdown.config.ts. Run pnpm install.
3. Turn the CLI command into an adapter
   • File to modify: packages/cli/src/commands/sync.ts.
   • Keep CONFLICT_MODES, ConflictMode, parseSyncConcurrency, SyncOptions, renderOutcomeLine, renderHumanSummary and createSyncAction. Re-export DEFAULT_SYNC_CONCURRENCY from @deepnote/cloud-sync so cli.ts needs no change. Replace ProjectSyncOutcome and SyncResult with the library types.
   • Replace syncWorkspace with an exported runSync(dir: string | undefined, options: SyncOptions): Promise<WorkspaceSyncResult> that does the following, in this order:
     1. Resolves rootDir = path.resolve(process.cwd(), dir ?? '.').
     2. Creates the root unless dryRun is set.
     3. Loads <root>/.env with dotenv.config.
     4. Resolves the token with resolveToken(options.token) and throws MissingTokenError when there is none.
     5. Picks the conflict policy:
        • Computes canPrompt = Boolean(process.stdin.isTTY && process.stdout.isTTY) && options.output === undefined, unchanged from today.
        • ask with canPrompt false (no TTY, or -o json even on a TTY) → 'skip', plus the existing debug('No interactive terminal; conflicts will be skipped. Use --on-conflict to decide up front.') line.
        • ask with canPrompt true → a function, also when dryRun is set; the library then treats it as 'skip', and no debug line is printed.
        • skip or override → that string.
     6. Calls the library's syncWorkspace with baseUrl: options.url ?? DEFAULT_API_URL and the remaining flags.
   • The ask function calls select with a message and an override choice label produced by a new private describeConflict(conflict: SyncConflict): { question: string; overrideLabel: string }. For each kind, describeConflict returns exactly the question and override label that the corresponding call site passes to resolveConflict today. For working-files-changed, the summary inside the question is files.map(f => `${f.path} (${f.reason})`).join(', '). The skip choice stays Skip this project for now.
   • The onEvent handler maps events as follows:
     • listing-projects → log(getChalk().dim(`Listing projects from ${baseUrl}…`)), only when options.output is unset.
     • project-outcome → log(renderOutcomeLine(outcome)), only when options.output is unset.
     • warning → warn(message).
     • file-transferred → debug, with the existing Downloaded … and Uploaded … text.
   • createSyncAction calls runSync. Its output and exit-code handling are unchanged; ExitPromptError still exits with code 1.
4. Move and convert the engine tests, and add adapter tests
   • Move packages/cli/src/commands/sync.test.ts to packages/cloud-sync/src/sync-workspace.test.ts with git mv and convert it in place. Test names and expected values stay unchanged:
     • Calls pass rootDir, baseUrl and token explicitly.
     • Prompt-driven tests pass an onConflict vi.fn instead of mocking select and stubbing TTYs; the Ctrl+C tests reject from it with the same error object. The converted prompt tests then cover the never-concurrent and hold-and-flush criteria.
     • Assertions on console.log and console.error become assertions on the collected SyncEvents, with the same message strings.
     • Delete the tests whose subject is TTY detection, the select choices, or .env and token handling; the CLI adapter tests cover that behavior.
   • Move fflate (the suite's archive fixtures) from devDependencies in packages/cli/package.json to devDependencies in packages/cloud-sync/package.json, never dependencies; nothing else in the CLI imports it.
   • Add to sync-workspace.test.ts:
     • In a dry run, a function policy is never called and both conflicting projects report skipped-conflict.
     • A .env file with DEEPNOTE_TOKEN in rootDir leaves process.env unchanged.
     • A started file replacement completes after cancellation: with allFiles: true and concurrency: 2, project A changed on both sides and project B pushing new working files one.csv and two.csv, hold B's DELETE of one.csv until A's function policy rejects. syncWorkspace rejects with the policy's error; one.csv is deleted and uploaded exactly once; two.csv is neither; the saved manifest lists one.csv in B's files with no pendingFileUploads.
     • concurrency values 0, -1, 0.5 and NaN reject with RangeError, with no fetch call and no rootDir created.
     • With the diverged-file fixture, allFiles: true and a function policy, the function receives { kind: 'working-files-changed', projectId: 'p1', projectName: 'Alpha', projectDir: 'Alpha', files: [{ path: '_deepnote_static/index.html', reason: 'changed in Deepnote' }] }.
   • Create packages/cli/src/commands/sync.test.ts. Mock only syncWorkspace from @deepnote/cloud-sync, with a vi.fn that drives onConflict and onEvent. Write expected output as literals, never produced by renderHumanSummary or outputJson. Cover:
     • Policy: skip and override pass through. ask with TTYs passes a function that, for each of the five conflict kinds, calls select with today's literal question and override label and returns the answer. ask without a TTY, and ask with TTYs and -o json, pass 'skip' and print the No interactive terminal debug line. ask with TTYs in a dry run passes a function and prints no debug line.
     • Events: the listing-projects and project-outcome lines print without -o and are suppressed with -o json; a warning reaches stderr verbatim.
     • Arguments: every flag reaches syncWorkspace, with dir as rootDir and --url as baseUrl. A token in <root>/.env is used, and --token overrides it.
     • createSyncAction: with no token anywhere it fails with exit code 2 and a message starting Missing authentication token., without calling syncWorkspace. -o json prints the result once as 2-space-indented JSON and no summary. Without -o, the summary, including the dry-run prefix and the untracked-files line, prints exactly as today. An unsuccessful result sets process.exitCode to 1 and the handler resolves; a successful one leaves it unset. An ExitPromptError from select fails with exit code 1 and the prompt's message.
5. Document the API
   • In packages/cloud-sync/README.md, add a syncWorkspace section with the option, event and conflict tables and the cancellation contract: a rejecting policy stops the run at its next checkpoints, so no new project or write starts; steps already under way (including a started file replacement) finish; a final manifest save is attempted unless it is a dry run; and the run then rejects with the same error, with no engine work still running. The option table states that concurrency must be a positive integer and that any other value rejects with RangeError. The conflict table describes projectDir and files[].path with the same wording as step 1.
   • Remove planProjectPaths, pathsOverlap, PlannedProjectPaths, sha256, baselineDiverged, assertNoSymbolicLinkAncestors, saveSyncManifest, isSafeRelativeFilePath and projectFilesDir from packages/cloud-sync/src/index.ts when grep finds no importer of them outside packages/cloud-sync.

Follow-ups

• Single-project sync (Get deepnote sync and publish ready for the editor and agents #543). Adds projectIds to syncWorkspace and deepnote sync --project <id|path>. Depends on this issue.
• MCP sync tool (Add sync, publish and app tools to the local MCP server #544). Adds deepnote_sync to @deepnote/mcp, built on syncWorkspace. Depends on this issue.