Allow finer control over execution of tasks

I would like to discuss about a way to improve the usability of tasks. In short, I think it would be great to have control over when (and if) a task is executed.

Related to #196 and #238.

Yes, I think that it should work similar to migrations: pre and post. We could even have pre-copy, pre-update, post-copy, post-update. That should be enough, right?

@baurmatt

@baurmatt had a good idea at #238 (comment): add $OPERATION={copy,update} to the environment. With that, you can simply include any needed if in your task code and perform it or not, without needing to overcomplicate the copier.yml file. Also it'd be fully backwards compatible.

IMHO the path to follow.

pre-copy, pre-update, post-copy, post-update

I'd love to have those indeed. Would this be accomplished with both STAGE and OPERATION env vars then? The four combinations would be mutually exclusive, meaning each one is only executed when the conditions are met.

`STAGE` / `OPERATION`	copy	update
before
after

Yes, well... definitely this couldn't be backwards-compatible after all.

I'm OK with implementing this incrementally if you'd like to make it backwards-compatible first, then add functionality like more stages.

@yajo

Hey @yajo, I'd like to start tackling this. Basically, I want to add an OPERATION environment variable when running tasks, as was discussed previously.

Do you already have an idea of how this could be implemented?
I was wondering if it would make sense to set the environment variable with os.environ["OPERATION"] = "update" to later on pass it in the extra_env argument:

run_tasks(
        conf, render, [{"task": t, "extra_env": {"STAGE": "task", "OPERATION": os.environ["OPERATION"]}} for t in conf.tasks]
    )

...but I'm not sure because it's like using a global variable 😅
Also, maybe we would not even need to pass it in the extra_env argument since we set it with os.environ?

What do you think?

What if we apply a logic like the one explained at #326 but for tasks too?

Isn't it exactly like what is discussed here 🙂 ? Passing everything needed (env vars) to tasks when executing them?

Lol yes...

I've got pretty clear now that copier 6 will break all things. It's gonna be fun 😆

@bswck

@bswck That would only work for Python scripts, but tasks can use executables written in any language.

Why? I just showed an example of invoking the GitHub CLI.

If you're referring to rendering scripts and then running them, session.run could provide facilities for it.

A Python layer for registering callbacks would not only make things easier, but also extend the overall ability to control tasks.

If you are, however, opposing the idea of having to define a Python script, I would argue then that the potential benefit justifies the trade-off.

@bswck

@bswck, @sisp meant that providing this feature through Python decorators like after_copy would make the feature, well, Python-only. It would require to write a Python script to run a Bash task or run any other executable that is not Python. A language-agnostic implementation would probably use environment variables instead of Python decorators, since you check env vars in any language and act accordingly. Then only, Copier could maybe provide such decorators to facilitate writing tasks in Python 🙂 So your idea is not incompatible with the feature 👍

FWIW I was scared away from yeoman because I didn't want to write js just for a template. In the same fashion I don't want to scare away js devs by requiring writing hooks in Python. The truth is that we can just use env variables and command calls, and that just gives the dev all possibilities. Copier has remained as lang-agnostic as possible and I wouldn't like to loose that.

in case it's helpful for others, I am testing this workaround which seems to work to only execute the tasks during the first execution of copier to initialize my project (create the required gitlab repos and extra stuff I don't want to repeat when doing copier update)

in copier.yml :

_tasks:
  - bash .project_init.sh

and in .project_init.sh :

#!/bin/bash

if [[ $PWD == /tmp* ]]; then
    echo "We are /tmp/. This is copier update. Exiting"
    exit 0
fi

if [ -d .git/ ]; then
  echo "This project is already initialized. Exiting"
  exit 0
fi

git init
git add .
git commit -a -m 'First commit by copier'

# here more initilization stuff

Script .project_init.sh` is another template in my copier project.

I need to do more testing but so far it seems to work for me

I just saw the 'good for newbies' label and could potentially set aside some time to work on this. Has there been any progress yet, or would it pretty much be a clean slate to work from

@sevdesk-ryanp

@sevdesk-ryanp thanks for offering your help! There is an unfinished PR here: #1511, if you want to take a look (and maybe take over).

Another workaround I have been using to avoid running intialization tasks on copier update uses an extensions.py file to define an is_copier_update variable:

# copier.yml

_jinja_extensions:
  - copier_templates_extensions.TemplateExtensionLoader
  - extensions.py:TemplateDefaultExtension
...

_tasks:
  - command: python setup_github.py  # Or any other "initialization-only" tasks
    when: "{{ not is_copier_update }}"

And then the extension defines is_copier_update:

# extensions.py

from jinja2.ext import Extension
from pathlib import Path

class TemplateDefaultExtension(Extension):
    def __init__(self, environment):
          super().__init__(environment)
          environment.globals["is_copier_update"] = Path(".copier-answers.yml").exists()

The .copier-answers.yml file will only exist in project root before invoking copier if the project was already made with copy, thus it signals when a copier update is happening.

This will require the --trust flag since it uses extensions, but I believe you already would have to using tasks. See extensions.

For small tasks I use this:

_tasks:
  - test "${PWD#'/tmp'}" = "$PWD" && git init # on first time only

I don't know if this goes here. I was looking for a way to populate some of the defaults by "proding" the system.

Either directly in the questions, or as a task recording variables to be used during the questions.

email:
  type: str
  help: What is your email address?
  default_cmd:
    command: git config user.email

@KalleDK

@KalleDK You can create a Jinja extension for this. See, e.g., https://github.com/pawamoy/copier-pdm/blob/adff9b64887d0b4c9ec0b42de1698b34858a511e/extensions.py#L23-L27 & https://github.com/pawamoy/copier-pdm/blob/adff9b64887d0b4c9ec0b42de1698b34858a511e/copier.yml#L11.

@KalleDK

@KalleDK You can create a Jinja extension for this. See, e.g., https://github.com/pawamoy/copier-pdm/blob/adff9b64887d0b4c9ec0b42de1698b34858a511e/extensions.py#L23-L27 & https://github.com/pawamoy/copier-pdm/blob/adff9b64887d0b4c9ec0b42de1698b34858a511e/copier.yml#L11.

Thank you - works like a charm.. Just needed to inject the extension loader :)

I don't think this issue should be closed. Seems like #1733 didn't fully resolve this issue.

The good thing is that the good work in #1510 and #1733 paved the way to implement a PR for this more easily.

#1510 gave Tasks the ability to use the _stage variable in its when condition, which is of value "before"/"after".
However it didn't set those stage values for tasks, only for migrations. Additionally, in the main:Worker.run_copy function, _execute_tasks was only called once after _render_template has been called, making tasks currently only a "post-copy" feature.

As a result, a way to resolve this issue would be:

make the Template.tasks function includes a stage parameter like in Template.migration_tasks,
then in main.py calls _execute_tasks twice, before and after the call the _render_template

However this has the implication that Phase.TASKS is kinda no longer valid, because there is no singular "Tasks" phase - there is now pre-copy tasks, and post-copy tasks. Also I am not sure how this would work with the _copier_phase variable

feat: add extension to add var to enable only running on initial

Navigation Menu

Repository navigation

Allow finer control over execution of tasks #240

16 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Allow finer control over execution of tasks #240

Description

Activity

yajo commented on Aug 11, 2020

yajo commented on Aug 11, 2020

pawamoy commented on Aug 11, 2020

yajo commented on Aug 11, 2020

pawamoy commented on Aug 11, 2020

pawamoy commented on Oct 6, 2020

yajo commented on Jan 17, 2021

pawamoy commented on Jan 17, 2021

yajo commented on Jan 17, 2021

16 remaining items

johnslavik commented on Jan 29, 2024

pawamoy commented on Jan 29, 2024

yajo commented on Jan 30, 2024

pescobar commented on Mar 12, 2024

sevdesk-ryanp commented on Jun 5, 2024

pawamoy commented on Jun 5, 2024

eckelsjd commented on Aug 27, 2024

woile commented on Sep 16, 2024

KalleDK commented on Nov 25, 2024

sisp commented on Nov 25, 2024

KalleDK commented on Nov 25, 2024

bhuynhdev commented on Mar 19, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions