Visitar URL original
SOS! Help! I cannot get enroot/pyxis to work: error: pyxis: error: a container with name "pyxis_nemo-25.07.gpt_oss" does not exist, and --container-image is not set · Issue #249 · NVIDIA/enroot · GitHub
Skip to content

SOS! Help! I cannot get enroot/pyxis to work: error: pyxis: error: a container with name "pyxis_nemo-25.07.gpt_oss" does not exist, and --container-image is not set #249

Open
Open
SOS! Help! I cannot get enroot/pyxis to work: error: pyxis: error: a container with name "pyxis_nemo-25.07.gpt_oss" does not exist, and --container-image is not set#249

Description

@tjoymeed
tjoymeed
opened on Sep 25, 2025

Hi there,

SOS! Could you please kindly help me? I just couldn't get pyxis work:

Thanks!

enroot import 'docker://nvcr.io#nvidia/nemo:25.07.gpt_oss'

enroot create --name pyxis_nemo-25.07.gpt_oss nvidia+nemo+25.07.gpt_oss.sqsh

And then since then I updated one of the sub-folders inside the container slightly by using the latest repo main branch, so I am not willing to directly to pull from NVIDIA registry in srun.

mike@ln01:~/enroot$ enroot list
cache
data
nemo-25.04
pyxis_hiyouga
pyxis_nemo-25.07.gpt_oss
pyxis_nemo-25.07.gpt_oss.sqsh
mike@ln01:~/enroot$ enroot version
3.5.0
mike@ln01:~/enroot$ ll
total 15122412
drwxr-xr-x 2 mike domain users        4096 Sep 25 17:05 ./
drwx------ 2 mike domain users        4096 Sep 25 16:43 ../
drwx------ 2 mike domain users        4096 Jun 30 11:48 cache/
drwx------ 2 mike domain users        4096 Sep 25 16:21 data/
drwxr-xr-x 2 mike domain users        4096 Jul 14 01:33 nemo-25.04/
drwxr-xr-x 2 mike domain users        4096 May 11 03:21 pyxis_hiyouga/
drwxrwxrwx 2 mike domain users        4096 Sep 21 17:57 pyxis_nemo-25.07.gpt_oss/
-rw-r----- 1 mike domain users 15485349888 Sep 24 22:53 pyxis_nemo-25.07.gpt_oss.sqsh


**mike@ln01:~/enroot$ srun --partition=gpu-interactive --account=gpu   --container-name="nemo-25.07.gpt_oss"   /bin/hostname**
srun: job 1276806 queued and waiting for resources
srun: job 1276806 has been allocated resources
slurmstepd-gn12: error: pyxis: error: a container with name "pyxis_nemo-25.07.gpt_oss" does not exist, and --container-image is not set
slurmstepd-gn12: error: spank: required plugin spank_pyxis.so: task_init() failed with rc=-1
slurmstepd-gn12: error: Failed to invoke spank plugin stack
srun: error: gn12: task 0: Exited with exit code 1



mike@ln01:~/enroot$ ll
total 15122412
drwxr-xr-x 2 mike domain users        4096 Sep 25 17:26 ./
drwx------ 2 mike domain users        4096 Sep 25 16:43 ../
drwx------ 2 mike domain users        4096 Jun 30 11:48 cache/
drwx------ 2 mike domain users        4096 Sep 25 16:21 data/
drwxr-xr-x 2 mike domain users        4096 Jul 14 01:33 nemo-25.04/
drwxr-xr-x 2 mike domain users        4096 May 11 03:21 pyxis_hiyouga/
drwxrwxrwx 2 mike domain users        4096 Sep 21 17:57 pyxis_nemo-25.07.gpt_oss/
-rw-r----- 1 mike domain users 15485349888 Sep 24 22:53 pyxis_nemo-25.07.gpt_oss.sqsh
mike@ln01:~/enroot$ echo $ENROOT_DATA_PATH 
/home/mike/enroot

Activity

  1. krono commented on Sep 25, 2025

    @krono
    krono
    on Sep 25, 2025
    Contributor

    This is not how the enroot/pyxis interaction will work.
    If you want an enroot container that is also avaiable via pyxis, you have to lookup how pyxis is configured in your cluster.

  2. tjoymeed commented on Sep 25, 2025

    @tjoymeed
    tjoymeed
    on Sep 25, 2025
    Author

    What do you mean?

    Note the following had no problem:

    mike@ln01:~/enroot$ srun --nodes=1 --ntasks=1 --partition=gpu-interactive --account=gpu --container-image=docker://ubuntu:22.04 /bin/echo "Pyxis works"

    srun: job 1276824 queued and waiting for resources

    srun: job 1276824 has been allocated resources

    pyxis: importing docker image: docker://ubuntu:22.04

    pyxis: imported docker image: docker://ubuntu:22.04

    Pyxis works

  3. krono commented on Sep 25, 2025

    @krono
    krono
    on Sep 25, 2025
    Contributor

    It does, but pyxis configures a lot of $ENROOT vars differenly, you should ask there probably…

  4. tjoymeed commented on Sep 25, 2025

    @tjoymeed
    tjoymeed
    on Sep 25, 2025
    Author

    I have to get the local updated unpacked rootfs folder working (since I did the enroot import and then enroot create and then did some local updates already...)

    I have also made the unpacked rootfs folder into a squashfs file:

    Create a squashfs file from your unpacked rootfs

    mksquashfs pyxis_nemo-25.07.gpt_oss pyxis_nemo-25.07.gpt_oss.sqsh -noappend

    when I use this image file, it hung:

    mike@ln01:~/enroot$ srun --nodes=1 --ntasks=1 --partition=gpu-interactive --account=gpu --container-image=/home/mike/enroot/pyxis_nemo-25.07.gpt_oss.sqsh /bin/echo "Pyxis works"
    srun: job 1276833 queued and waiting for resources
    srun: job 1276833 has been allocated resources

    [* HUNG! I had to type CRTL+C to stop it *]

    ^Csrun: interrupt (one more within 1 sec to abort)
    srun: StepId=1276833.0 task 0: running
    ^Csrun: sending Ctrl-C to StepId=1276833.0
    srun: forcing job termination
    srun: Job step aborted: Waiting up to 62 seconds for job step to finish.
    slurmstepd-gn12: error: *** STEP 1276833.0 ON gx12 CANCELLED AT 2025-09-25T18:09:16 ***
    slurmstepd-gn12: error: pyxis: child 1070021 terminated with signal 9
    ^Csrun: sending Ctrl-C to StepId=1276833.0
    srun: job abort in progress

    I recall that when I created it, it had some warning message:
    Not sure if this warning message is the problem:

    Parallel mksquashfs: Using 96 processors
    Creating 4.0 filesystem on pyxis_nemo-25.07.gpt_oss.sqsh, block size 131072.
    Could not open pyxis_nemo-25.07.gpt_oss/etc/slurm, skipping...

  5. tjoymeed commented on Sep 25, 2025

    @tjoymeed
    tjoymeed
    on Sep 25, 2025
    Author

    It does, but pyxis configures a lot of $ENROOT vars differenly, you should ask there probably…

    Thanks! Okay I added: export ENROOT_RUNTIME_PATH="$HOME/.local/run/enroot"

    But still got the same issues...

  6. krono commented on Sep 25, 2025

    @krono
    krono
    on Sep 25, 2025
    Contributor

    That is expected. pyxis actively ignores most $ENROOT_ variables.

    Maybe this will work?

    enroot import ....
enroot create ...
enroot export ... image.sqsh
mv image.sqsh $GLOBAL_STORAGE
srun ... --container-image=$GLOBAL_STORAGE/image.sqsh ...

    point is the export.

  7. tjoymeed commented on Sep 25, 2025

    @tjoymeed
    tjoymeed
    on Sep 25, 2025
    Author

    Thanks a lot!

    Yes, the enroot export is better than the mksquashfs I've used yesterday, which had a warning:

    Today enroot export is cleaner, with no warning.

    Still it hung, what's the problem?

    mike@ln01:~/enroot$ ll
    total 15122412
    drwxr-xr-x 2 mike domain users 4096 Sep 25 17:26 ./
    drwx------ 2 mike domain users 4096 Sep 25 18:24 ../
    drwx------ 2 mike domain users 4096 Jun 30 11:48 cache/
    drwx------ 2 mike domain users 4096 Sep 25 16:21 data/
    drwxr-xr-x 2 mike domain users 4096 Jul 14 01:33 nemo-25.04/
    drwxr-xr-x 2 mike domain users 4096 May 11 03:21 pyxis_hiyouga/
    drwxrwxrwx 2 mike domain users 4096 Sep 21 17:57 pyxis_nemo-25.07.gpt_oss/
    -rw-r----- 1 mike domain users 15485349888 Sep 24 22:53 pyxis_nemo-25.07.gpt_oss.sqsh

    mike@ln01:~/enroot$ enroot export --output test.sqsh pyxis_nemo-25.07.gpt_oss
    [INFO] Creating squashfs filesystem...

    Parallel mksquashfs: Using 96 processors
    Creating 4.0 filesystem on /sc/home/mike/enroot/test.sqsh, block size 131072.
    [===============================================================================================================================================================================================================] 529472/529472 100%

    Exportable Squashfs 4.0 filesystem, lzo compressed, data block size 131072
    uncompressed data, compressed metadata, compressed fragments,
    compressed xattrs, compressed ids
    duplicates are removed
    Filesystem size 29009641.95 Kbytes (28329.73 Mbytes)
    89.59% of uncompressed filesystem size (32380218.35 Kbytes)
    Inode table size 4553800 bytes (4447.07 Kbytes)
    30.98% of uncompressed inode table size (14698293 bytes)
    Directory table size 4437568 bytes (4333.56 Kbytes)
    38.26% of uncompressed directory table size (11597683 bytes)
    Number of duplicate files found 75371
    Number of inodes 377279
    Number of files 310435
    Number of fragments 20631
    Number of symbolic links 21513
    Number of device nodes 0
    Number of fifo nodes 0
    Number of socket nodes 0
    Number of directories 45331
    Number of hard-links 6
    Number of ids (unique uids + gids) 1
    Number of uids 1
    root (0)
    Number of gids 1
    root (0)
    mike@ln01:/enroot$
    mike@ln01:    /enroot$
    mike@ln01:~/enroot$ ll
    total 44132056
    drwxr-xr-x 2 mike domain users 4096 Sep 25 18:40 ./
    drwx------ 2 mike domain users 4096 Sep 25 18:24 ../
    drwx------ 2 mike domain users 4096 Jun 30 11:48 cache/
    drwx------ 2 mike domain users 4096 Sep 25 16:21 data/
    drwxr-xr-x 2 mike domain users 4096 Jul 14 01:33 nemo-25.04/
    drwxr-xr-x 2 mike domain users 4096 May 11 03:21 pyxis_hiyouga/
    drwxrwxrwx 2 mike domain users 4096 Sep 21 17:57 pyxis_nemo-25.07.gpt_oss/
    -rw-r----- 1 mike domain users 15485349888 Sep 24 22:53 pyxis_nemo-25.07.gpt_oss.sqsh
    -rw-r----- 1 mike domain users 29705875456 Sep 25 19:14 test.sqsh

    mike@ln01:~/enroot$ srun --nodes=1 --ntasks=1 --partition=gpu-interactive --account=gpu --container-image=/home/mike/enroot/test.sqsh /bin/echo "Pyxis works"

    srun: job 1277196 queued and waiting for resources
    srun: job 1277196 has been allocated resources

    [HUNG HERE! I HAD TO CTRL+C]

    ^Csrun: interrupt (one more within 1 sec to abort)
    srun: StepId=1277196.0 task 0: running
    ^Csrun: sending Ctrl-C to StepId=1277196.0
    srun: forcing job termination
    srun: Job step aborted: Waiting up to 62 seconds for job step to finish.
    slurmstepd-gn11: error: *** STEP 1277196.0 ON gn11 CANCELLED AT 2025-09-25T19:18:52 ***
    slurmstepd-gn11: error: pyxis: child 1231834 terminated with signal 9

  8. tjoymeed commented on Sep 25, 2025

    @tjoymeed
    tjoymeed
    on Sep 25, 2025
    Author

    That is expected. pyxis actively ignores most $ENROOT_ variables.

    Maybe this will work?

    enroot import ....
enroot create ...
enroot export ... image.sqsh
mv image.sqsh $GLOBAL_STORAGE
srun ... --container-image=$GLOBAL_STORAGE/image.sqsh ...

    point is the export.

    Thanks a lot!

    My home directory is shared anyway on the compute node. Is it the $GLOBAL_STORAGE you meant?

  9. krono commented on Sep 25, 2025

    @krono
    krono
    on Sep 25, 2025
    Contributor

    Yes.
    Anyway, further problems should probably reportet at https://github.com/NVIDIA/pyxis

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions