[6.x] Importing - #19612

It imports data from JSON, CSV or XML sources (local files or URLs) into Craft CMS elements or Eloquent models.

Core concepts:

• Data Types => the data sources this feature can work with
• Transformers => normalise and manipulate incoming data, and sometimes map it to existing “slots”
• Importers => do the heavy lifting. Each importer knows how to write data into one specific target (e.g. Entries, Assets, Users, System Messages)
• Field import handlers => optional, import-specific handling for a field type (e.g. creating assets from files referenced in an Assets field)
• Import Plans => a named, ordered list of steps. Each step pairs an importer with its own source, transformer, batch size, settings and mapping

Extensibility:

• you can register extra data types, importer types and field import handlers
• you can register non-editable import plans in code from a service provider
• you can manipulate your data any way you want via transformers
• you can add a CLI command for your own importer by extending CraftCms\Cms\Import\Commands\ImportCommand

Data Types:

• there are 3 built-in data types: JSON, XML and CSV
• extra types can be registered from a service provider’s boot() via CraftCms\Cms\Import\DataTypes\DataTypes::register(). To replace a built-in one, remove() it first

Transformers:

• there are 3 built-in transformers, all related to element importing:
  • ElementTransformer => the base element transformer, and the default for element importers without their own (in core, Users)
  • EntryTransformer => default when importing Entries
  • AssetTransformer => default when importing Assets
• ElementTransformer auto-matches incoming keys to attributes and field handles even when the names aren’t exact (e.g. Plain Text / plain_text → plainText), and supports convention-based normalize{PropName}()
  methods
• you can create your own transformers and use them to manipulate the data however you wish. For example, you can uppercase every value imported into myPlainTextField, or run complex checks to decide what value
  myOtherField gets. A transformer can be given as an instance or a class name

Importers:

• each importable target has its own concrete importer class. Being importable means having an importer registered:
  • ElementImporter (abstract) => EntryImporter, AssetImporter, UserImporter. Each implements targetClass(). Common settings: site (required) and fieldLayout
    • EntryImporter adds section and entryType, and can set an entry’s parent (parentId) in Structure sections
    • AssetImporter adds volume
    • UserImporter won’t import sensitive attributes like admin or password; the only sensitive ones it allows are affiliatedSiteId, email, passwordResetRequired and photoId. photoId can also be a URL or a
      local path to the photo file
  • ModelImporter (abstract) => SystemMessageImporter is the only built-in. Its mappable destinations are the model table’s columns
• Address has no importer, so it’s never a standalone target. It can only be imported as nested content (Addresses field / User addresses)
• each importer defines its own settings form and validation rules
• every step has a batch size (default 100, max 1000, 0 disables batching)
• extra importer types (e.g. for plugin element types or models) can be registered from a service provider’s boot() via CraftCms\Cms\Import\ImporterTypes::register()

Field import handlers:

• import-specific work for a field type can live in a class implementing CraftCms\Cms\Import\FieldHandlers\FieldImportHandlerInterface, so the field itself needs no import code
• a handler normalizes the incoming value, can offer extra per-field settings in the mapping UI, and can queue side effects that should only happen once the item has been imported ($importer->afterItemImported())
• register one from a service provider’s boot() via CraftCms\Cms\Import\FieldHandlers\FieldImportHandlers::register(). A field uses the handler for its own class or its nearest parent class
• the built-in AssetsFieldImportHandler lets an Assets field value mix asset IDs with URLs and local file paths, creating assets from the files in the field’s upload location
  • the fileConflict setting decides what happens when a file with the same name already exists: use the existing asset (default), replace its file, or create a new asset
  • local paths must be within Craft’s temp folder, or within the project root but outside the storage folder. Dotfiles, dot-prefixed folders and restricted directories aren’t allowed
  • local paths are refused altogether when the step’s source is remote

Import plans:

• there are 2 kinds of import plans:
  • editable => created via the Control Panel and stored in the database (import_plans)
  • non-editable => registered in code from a service provider’s boot() via CraftCms\Cms\Import\ImportPlan::register('handle', fn () => new ImportPlanData()...), and committed to your repository. The factory only
    runs when the plans are first loaded, so registering them adds no work at boot and doesn’t stop artisan config:cache from working. The registered handle becomes the plan’s handle. A plan that is invalid, or that sets a
    different handle, is skipped and logged. Registering the same handle twice throws; remove() unregisters a plan
• for each step you:
  • choose the importer and fill in its settings. For example, for an Entry importer, you pick the site, section and entry type
  • specify the source that holds the data you want to import: a @root-relative or aliased path, or a URL
  • optionally choose a transformer and a batch size
  • define which importable attribute and field maps to which incoming value (map), or fill one from a template instead (templates)
  • choose which values should be used to match against existing elements (matchCriteria)
  • choose which values should be cleared out if they’re empty or missing from the incoming data (clearableItems)
  • choose whether you’d like to keep nested content that’s missing from the incoming data (keepMissingNestedElements)
• URL sources are validated before they’re fetched. URLs resolving to private IP addresses are rejected unless their host is listed in the allowedPrivateHosts config setting; loopback, link-local and cloud metadata
  addresses are always rejected. The downloaded file’s contents must match its data type
• steps run sequentially, and each step runs as its own batch, so one bad step doesn’t kill the whole plan. Per-item failures are logged and skipped, not fatal
• when a step’s source has more items than its batch size, the first batch job reads it once and stores all its items on the importDisk (a local temp folder by default; use a shared disk if queue jobs can run on
  different
  servers), and each later job reads only its own items. They’re cleaned up when the step finishes, and stale ones are removed by garbage collection. Otherwise (batching disabled, or a job that stops early on a source that
  fits in one batch) a follow-up job reads the source again
• batch jobs aren’t retried (that would import their items again), and a job hands the rest of its items to the next one if it gets close to the memory limit or timeout
• editable plans can be duplicated and reordered

Mapping UI

• a mapping table with Destination / Incoming data / Match / Clear columns. Container fields (Matrix, Addresses, Content Block) and container properties (User addresses) open a nested slideout, which can nest to any
  depth
• The incoming-data select is a combobox that shows the value from the source’s first row next to each option, plus “Please select” (don’t import) and “Use template”.
• “Use template” shows a Template field that takes plain text or an object template, like an entry type’s Title Format. Typing { lists the source’s columns as variables ({title}, or {{ object['First Name'] }} for names that aren’t identifiers). Templates work inside nested mappings too, where {blocks.heading} is the current block’s value. They’re rendered in Twig’s sandbox, since saving import plans doesn’t
  require admin, and checked when the mapping is applied and when the plan is saved
• matchCriteria, clearableItems, and keepMissingNestedElements are reserved: incoming data using them, at any depth, is never offered as a column and can’t be read by a template
• the most likely incoming column is pre-selected for each destination
• attributes marked canBeSet: false (e.g., id, uid) can only be used for matching, not set

Match criteria — three tiers, last wins

1. In the data — a matchCriteria key on the row, [systemHandle => incomingKey].
   Also looks inside $data['fields']. If the row has no value for that key, the value given is matched on as-is.
2. Step-level — from the UI or a registered import plan, which is read once per step. It names the keys to match on; each row supplies the values, and a row with no value for a key isn’t matched on it.
3. Transformer — additionalMatchCriteria(), already resolved to actual values.

Values from tiers 1 and 2 are matched literally (query syntax like * or not 5 isn’t applied); tier 3 values are used as given.

Clearable items

An array shaped like the match criteria. If a field or attribute is

• marked clearable and its value is missing or empty in the incoming data, it’s forced to null and explicitly applied
• not marked clearable and its value is empty in the incoming data, its key is unset() entirely, so the existing value is left untouched.

Importing:

• you can trigger an import via the UI: CP > Import > Run, for editable and registered plans alike (requires triggerImportPlans)
• or via the CLI, with one command per importer ({source} is a @root-relative or aliased path, or a URL):
  • artisan craft:import:entries {source} [--site=] [--section=] [--entry-type=] [--transformer=] [--match-criteria=]
  • artisan craft:import:assets {source} [--site=] [--volume=] …
  • artisan craft:import:users {source} …
  • artisan craft:import:system-messages {source} …
  • any options you leave out are prompted for (--site only on multisite), and --match-criteria takes JSON (invalid JSON is silently ignored)

Making things importable:

• Elements & models: register a concrete ElementImporter / ModelImporter subclass via ImporterTypes::register()
• CraftCms\Cms\Support\Attributes\Importable marks an element’s properties as importable. Its flags are excludeFromUiMapping, isContainer, canBeMatchCriteria, canBeCleared and canBeSet
• CraftCms\Cms\Element\Contracts\ImportableContainerPropertiesInterface must be implemented by an element type with a container property (isContainer: true), like User’s addresses
• CraftCms\Cms\FieldLayout\Contracts\ImportableFieldLayoutElementInterface makes field layout elements importable
• CraftCms\Cms\Field\Contracts\ImportableElementContainerFieldInterface lets container fields be imported correctly
• CraftCms\Cms\Import\FieldHandlers\FieldImportHandlerInterface adds import-specific handling for a field type

Events:

ItemImporting / ItemImported, ImportPlanSaving / ImportPlanSaved, ImportDispatching / ImportDispatched. The *ing events are cancellable; to cancel, a listener sets $event->isValid = false. ItemImported
also carries the element or model the data was imported into.

Each import run also fires lifecycle events, all sharing the run’s runId: ImportStarted, ImportStepStarted, ImportChunkStarted / ImportChunkFinished, ImportStepFinished and ImportFinished (the *Finished
events report whether anything failed). CLI imports fire these too, apart from the chunk events.

Config:

• importDisk => the filesystem disk import runs store their parsed source data on (set it to a shared disk if queue jobs can run on different servers)
• allowedPrivateHosts => hostnames a source URL may resolve to private IP addresses for

Permissions:

There’s a permissions group import with viewImportPlans and, nested under it, saveImportPlans, deleteImportPlans and triggerImportPlans.

• running import plans can create and update elements and other data regardless of the user’s other permissions, which the permission’s warning points out
• users who can view but not save import plans see the plans and their mapping read-only, and can only open the mapping for a saved step’s source, so they can’t make the server fetch a source of their choosing

Related issues

CMS-2390