docker-machine env xxx hangs forever #1500

Activity

1. pedroxs commented on Jul 10, 2015

   

   pedroxs

   on Jul 10, 2015

   More actions

   I have similar issue with windows 7 OS. here is a gist of the log file.
2. CpuID commented on Jul 11, 2015

   

   CpuID

   on Jul 11, 2015

   More actions

   I am noticing "docker-machine env machinename" hang a lot also, especially when it is in my ~/.profile, causes new terminals to hang and not start 50% of the time approx. Currently just ctrl+c'ing the new tab shell, and spawning another one 2 seconds later tends to do the job.

   Difficult to reproduce in a new/active terminal, tends to be far more often when spawning a new one.

   OSX 10.10.4, Docker Machine 0.3.0. These issues were not present in 0.2.x.
3. a0s commented on Jul 13, 2015

   

   a0s

   on Jul 13, 2015

   More actions

   I have same issue with strange exit status 255

   docker-machine -D env docker
   shell: bash
   executing: /Applications/VMware Fusion.app/Contents/Library/vmrun list
   MAC address in VMX: 00:0c:29:b6:95:bb
   IP found in DHCP lease table: 172.16.1.131
   invalid certs detected; regenerating for 172.16.1.131:2376
   command=configureAuth machine=docker
   executing: /Applications/VMware Fusion.app/Contents/Library/vmrun list
   MAC address in VMX: 00:0c:29:b6:95:bb
   IP found in DHCP lease table: 172.16.1.131
   Using SSH client type: external
   About to run SSH command:
   cat /etc/os-release
   &{/usr/bin/ssh [/usr/bin/ssh -o PasswordAuthentication=no -o IdentitiesOnly=yes -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -i /Users/orangeudav/.docker/machine/machines/docker/id_rsa -p 22 docker@172.16.1.131 cat /etc/os-release] []  <nil> <nil> <nil> [] <nil> <nil> <nil> <nil> false [] [] [] [] <nil>}
   <HAAAAAANGIIING HERE>
   SSH cmd err, output: exit status 255:
   Error getting SSH command: exit status 255
   export DOCKER_TLS_VERIFY="1"
   export DOCKER_HOST="tcp://172.16.1.131:2376"
   export DOCKER_CERT_PATH="/Users/orangeudav/.docker/machine/machines/docker"
   export DOCKER_MACHINE_NAME="docker"
   # Run this command to configure your shell:
   # eval "$(docker-machine env docker)"
   
4. chantra commented on Jul 14, 2015

   

   chantra

   on Jul 14, 2015

   Contributor

   More actions

   I am able to reproduce this issue on MacOSX for the virtualbox driver. The issue typically happens when the route to the private network is missing (the one used by docker cli).
   In my case, one of the reason I suspect this route is missing is because I am using a VPN that tends to mess up with priv network routes. A fix to this is to run something along:

   sudo route add -net 192.168.99.0/24 -interface vboxnet6
   

   Where 192.168.99.0/24 is the network range used by boot2docker and vboxnet6 the interface assigned to boot2docker private network.

   👍React with 👍4Reacted by Boris Cherny, Gerard Braad, Sylwester Lachiewicz and Mike Wabst
5. inkel commented on Jul 15, 2015

   

   inkel

   on Jul 15, 2015

   More actions

   I've tried @chantra's solution but didn't do the trick, docker-machine still hangs when in a new shell with any subcommand. Interrupting with ^C and trying again works, though.
6. nathanleclaire commented on Jul 15, 2015

   

   nathanleclaire

   on Jul 15, 2015

   Contributor

   More actions

   Hi all, anyone who is encountering this issue, one of the possible issues at play is that the Docker daemon is not running.

   Can you please try / paste the output of the following commands (this is for VMs running boot2docker):

   $ docker-machine -D ssh machinename sudo /etc/init.d/docker restart
   ...
   $ docker-machine -D env machinename
   ...

   Thanks!
7. inkel commented on Jul 15, 2015

   

   inkel

   on Jul 15, 2015

   More actions

   @nathanleclaire when I try to run that command, it hangs here:

   $ docker-machine -D ssh dev sudo /etc/init.d/docker restart
   executing: /usr/local/bin/VBoxManage showvminfo dev --machinereadable
   STDOUT: name="dev"
   groups="/"
   ostype="Linux 2.6 / 3.x (64-bit)"
   UUID="19ac89e0-98cf-49e8-b306-ca58878604a3"
   CfgFile="/Users/inkel/.docker/machine/machines/dev/dev/dev.vbox"
   SnapFldr="/Users/inkel/.docker/machine/machines/dev/dev/Snapshots"
   LogFldr="/Users/inkel/.docker/machine/machines/dev/dev/Logs"
   hardwareuuid="19ac89e0-98cf-49e8-b306-ca58878604a3"
   memory=2048
   pagefusion="off"
   vram=8
   cpuexecutioncap=100
   hpet="on"
   chipset="piix3"
   firmware="BIOS"
   cpus=4
   pae="on"
   longmode="on"
   synthcpu="off"
   bootmenu="disabled"
   boot1="dvd"
   boot2="dvd"
   boot3="disk"
   boot4="none"
   acpi="on"
   ioapic="on"
   biossystemtimeoffset=0
   rtcuseutc="on"
   hwvirtex="on"
   nestedpaging="on"
   largepages="on"
   vtxvpid="on"
   vtxux="on"
   VMState="running"
   VMStateChangeTime="2015-07-15T21:20:06.395000000"
   monitorcount=1
   accelerate3d="off"
   accelerate2dvideo="off"
   teleporterenabled="off"
   teleporterport=0
   teleporteraddress=""
   teleporterpassword=""
   tracing-enabled="off"
   tracing-allow-vm-access="off"
   tracing-config=""
   autostart-enabled="off"
   autostart-delay=0
   defaultfrontend=""
   storagecontrollername0="SATA"
   storagecontrollertype0="IntelAhci"
   storagecontrollerinstance0="0"
   storagecontrollermaxportcount0="30"
   storagecontrollerportcount0="30"
   storagecontrollerbootable0="on"
   "SATA-0-0"="/Users/inkel/.docker/machine/machines/dev/boot2docker.iso"
   "SATA-ImageUUID-0-0"="c5d1d610-61ba-4416-8721-f65383bd9595"
   "SATA-tempeject"="off"
   "SATA-IsEjected"="off"
   "SATA-1-0"="/Users/inkel/.docker/machine/machines/dev/disk.vmdk"
   "SATA-ImageUUID-1-0"="9719a1db-dfaf-441e-8d8f-47ee0b18293b"
   "SATA-2-0"="none"
   "SATA-3-0"="none"
   "SATA-4-0"="none"
   "SATA-5-0"="none"
   "SATA-6-0"="none"
   "SATA-7-0"="none"
   "SATA-8-0"="none"
   "SATA-9-0"="none"
   "SATA-10-0"="none"
   "SATA-11-0"="none"
   "SATA-12-0"="none"
   "SATA-13-0"="none"
   "SATA-14-0"="none"
   "SATA-15-0"="none"
   "SATA-16-0"="none"
   "SATA-17-0"="none"
   "SATA-18-0"="none"
   "SATA-19-0"="none"
   "SATA-20-0"="none"
   "SATA-21-0"="none"
   "SATA-22-0"="none"
   "SATA-23-0"="none"
   "SATA-24-0"="none"
   "SATA-25-0"="none"
   "SATA-26-0"="none"
   "SATA-27-0"="none"
   "SATA-28-0"="none"
   "SATA-29-0"="none"
   natnet1="nat"
   macaddress1="08002743999C"
   cableconnected1="on"
   nic1="nat"
   nictype1="virtio"
   nicspeed1="0"
   mtu="0"
   sockSnd="64"
   sockRcv="64"
   tcpWndSnd="64"
   tcpWndRcv="64"
   Forwarding(0)="ssh,tcp,127.0.0.1,54973,,22"
   hostonlyadapter2="vboxnet5"
   macaddress2="080027A3AA51"
   cableconnected2="on"
   nic2="hostonly"
   nictype2="82540EM"
   nicspeed2="0"
   nic3="none"
   nic4="none"
   nic5="none"
   nic6="none"
   nic7="none"
   nic8="none"
   hidpointing="ps2mouse"
   hidkeyboard="ps2kbd"
   uart1="off"
   uart2="off"
   lpt1="off"
   lpt2="off"
   audio="none"
   clipboard="disabled"
   draganddrop="disabled"
   SessionType="headless"
   VideoMode="720,400,0"@0,0
   vrde="off"
   usb="off"
   ehci="off"
   SharedFolderNameMachineMapping1="Users"
   SharedFolderPathMachineMapping1="/Users"
   VRDEActiveConnection="off"
   VRDEClients=0
   vcpenabled="off"
   vcpscreens=0
   vcpfile="/Users/inkel/.docker/machine/machines/dev/dev/dev.webm"
   vcpwidth=1024
   vcpheight=768
   vcprate=512
   vcpfps=25
   GuestMemoryBalloon=0
   GuestOSType="Linux26_64"
   GuestAdditionsRunLevel=1
   GuestAdditionsVersion="4.3.20 r96996"
   GuestAdditionsFacility_VirtualBox Base Driver=50,1436995222662
   GuestAdditionsFacility_Seamless Mode=0,1436995222662
   GuestAdditionsFacility_Graphics Mode=0,1436995222662
   
   STDERR:
   Using SSH client type: external
   About to run SSH command:
   sudo /etc/init.d/docker restart
   &{/usr/bin/ssh [/usr/bin/ssh -o PasswordAuthentication=no -o IdentitiesOnly=yes -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -i /Users/inkel/.docker/machine/machines/dev/id_rsa -p 54973 docker@localhost sudo /etc/init.d/docker restart] []  <nil> <nil> <nil> [] <nil> <nil> <nil> ?reflect.Value? false [] [] [] [] <nil>}
   

   Once if finishes (if it ever does) I'll paste what's next.
8. nathanleclaire commented on Jul 15, 2015

   

   nathanleclaire

   on Jul 15, 2015

   Contributor

   More actions

   @inkel Hm, it should run pretty much right away if it's going to succeed, so if it takes more than a few seconds something's wrong.
9. inkel commented on Jul 15, 2015

   

   inkel

   on Jul 15, 2015

   More actions

   It just finished with:

   SSH cmd err, output: <nil>:
   

   It took definitively more than a few seconds. I'm running again with time to see how much.
10. inkel commented on Jul 15, 2015

    

    inkel

    on Jul 15, 2015

    More actions

    $ docker-machine -D ssh dev sudo /etc/init.d/docker restart
    …same long output…
    real    5m1.197s
    user    0m0.045s
    sys 0m0.035s
    
11. nathanleclaire commented on Jul 15, 2015

    

    nathanleclaire

    on Jul 15, 2015

    Contributor

    More actions

    If you run :

    $ /usr/bin/ssh -vvv -o PasswordAuthentication=no -o IdentitiesOnly=yes -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -i /Users/inkel/.docker/machine/machines/dev/id_rsa -p 54973 docker@localhost sudo /etc/init.d/docker restart

    from the CLI on its own, what do you get?
12. inkel commented on Jul 15, 2015

    

    inkel

    on Jul 15, 2015

    More actions

    inkel@miralejos ~
    $ /usr/bin/ssh -vvv -o PasswordAuthentication=no -o IdentitiesOnly=yes -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -i /Users/inkel/.docker/machine/machines/dev/id_rsa -p 54973 docker@localhost sudo /etc/init.d/docker restart 2>&1 | pbcopy
    debug1: multiplexing control connection
    debug3: fd 7 is O_NONBLOCK
    debug3: fd 7 is O_NONBLOCK
    debug1: channel 1: new [mux-control]
    debug3: channel_post_mux_listener: new mux channel 1 fd 7
    debug3: mux_master_read_cb: channel 1: hello sent
    debug2: set_control_persist_exit_time: cancel scheduled exit
    debug3: mux_master_read_cb: channel 1 packet type 0x00000001 len 4
    debug2: process_mux_master_hello: channel 1 slave version 4
    debug3: mux_master_read_cb: channel 1 packet type 0x10000004 len 4
    debug2: process_mux_alive_check: channel 1: alive check
    debug3: mux_master_read_cb: channel 1 packet type 0x10000002 len 92
    debug2: process_mux_new_session: channel 1: request tty 0, X 1, agent 0, subsys 0, term "xterm", cmd "sudo /etc/init.d/docker restart", env 1
    debug3: process_mux_new_session: got fds stdin 8, stdout 9, stderr 10
    debug2: fd 9 setting O_NONBLOCK
    debug3: fd 10 is O_NONBLOCK
    debug1: channel 2: new [client-session]
    debug2: process_mux_new_session: channel_new: 2 linked to control channel 1
    debug2: channel 2: send open
    debug2: callback start
    debug2: client_session2_setup: id 2
    debug1: Sending environment.
    debug1: Sending env LANG = en_US.UTF-8
    debug2: channel 2: request env confirm 0
    debug1: Sending command: sudo /etc/init.d/docker restart
    debug2: channel 2: request exec confirm 1
    debug3: mux_session_confirm: sending success reply
    debug2: callback done
    debug2: channel 2: open confirm rwindow 0 rmax 32768
    debug2: channel 2: rcvd adjust 2097152
    debug2: channel_input_status_confirm: type 99 id 2
    debug2: exec request accepted on channel 2
    debug1: client_input_channel_req: channel 2 rtype exit-status reply 0
    debug3: mux_exit_message: channel 2: exit message, evitval 0
    debug1: client_input_channel_req: channel 2 rtype eow@openssh.com reply 0
    debug2: channel 2: rcvd eow
    debug2: channel 2: close_read
    debug2: channel 2: input open -> closed
    debug2: channel 2: rcvd eof
    debug2: channel 2: output open -> drain
    debug2: channel 2: obuf empty
    debug2: channel 2: close_write
    debug2: channel 2: output drain -> closed
    debug2: channel 2: rcvd close
    debug3: channel 2: will not send data after close
    debug2: channel 2: send close
    debug2: channel 2: is dead
    debug2: channel 2: gc: notify user
    debug3: mux_master_session_cleanup_cb: entering for channel 2
    debug2: channel 1: rcvd close
    debug2: channel 1: output open -> drain
    debug2: channel 1: close_read
    debug2: channel 1: input open -> closed
    debug2: channel 2: gc: user detached
    debug2: channel 2: is dead
    debug2: channel 2: garbage collecting
    debug1: channel 2: free: client-session, nchannels 3
    debug3: channel 2: status: The following connections are open:
      #2 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1)
    
    debug2: channel 1: obuf empty
    debug2: channel 1: close_write
    debug2: channel 1: output drain -> closed
    debug2: channel 1: is dead (local)
    debug2: channel 1: gc: notify user
    debug3: mux_master_control_cleanup_cb: entering for channel 1
    debug2: channel 1: gc: user detached
    debug2: channel 1: is dead (local)
    debug2: channel 1: garbage collecting
    debug1: channel 1: free: mux-control, nchannels 2
    debug3: channel 1: status: The following connections are open:
    
    debug2: set_control_persist_exit_time: schedule exit in 300 seconds
    inkel@miralejos ~
    $
    
13. inkel commented on Jul 15, 2015

    

    inkel

    on Jul 15, 2015

    More actions

    I think I've made a mistake with my previous comment, this is the right output:

    $ /usr/bin/ssh -vvv -o PasswordAuthentication=no -o IdentitiesOnly=yes -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=quiet -o ConnectionAttempts=3 -o ConnectTimeout=10 -i /Users/inkel/.docker/machine/machines/dev/id_rsa -p 54973 docker@localhost sudo /etc/init.d/docker restart 2>&1
    OpenSSH_6.2p2, OSSLShim 0.9.8r 8 Dec 2011
    debug1: Reading configuration data /Users/inkel/.ssh/config
    debug1: /Users/inkel/.ssh/config line 3: Applying options for *
    debug1: Reading configuration data /etc/ssh_config
    debug1: /etc/ssh_config line 20: Applying options for *
    debug1: auto-mux: Trying existing master
    debug1: Control socket "/Users/inkel/.ssh/master-docker@localhost:54973" does not exist
    debug2: ssh_connect: needpriv 0
    debug1: Connecting to localhost [::1] port 54973.
    debug2: fd 5 setting O_NONBLOCK
    debug1: connect to address ::1 port 54973: Connection refused
    debug1: Connecting to localhost [127.0.0.1] port 54973.
    debug2: fd 5 setting O_NONBLOCK
    debug1: fd 5 clearing O_NONBLOCK
    debug1: Connection established.
    debug3: timeout: 10000 ms remain after connect
    debug3: Incorrect RSA1 identifier
    debug3: Could not load "/Users/inkel/.docker/machine/machines/dev/id_rsa" as a RSA1 public key
    debug1: identity file /Users/inkel/.docker/machine/machines/dev/id_rsa type 1
    debug1: identity file /Users/inkel/.docker/machine/machines/dev/id_rsa-cert type -1
    debug1: Enabling compatibility mode for protocol 2.0
    debug1: Local version string SSH-2.0-OpenSSH_6.2
    debug1: Remote protocol version 2.0, remote software version OpenSSH_6.0
    debug1: match: OpenSSH_6.0 pat OpenSSH*
    debug2: fd 5 setting O_NONBLOCK
    debug3: put_host_port: [localhost]:54973
    debug3: load_hostkeys: loading entries for host "[localhost]:54973" from file "/dev/null"
    debug3: load_hostkeys: loaded 0 keys
    debug1: SSH2_MSG_KEXINIT sent
    debug1: SSH2_MSG_KEXINIT received
    debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
    debug2: kex_parse_kexinit: ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ssh-rsa,ssh-dss
    debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
    debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
    debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
    debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
    debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
    debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
    debug2: kex_parse_kexinit:
    debug2: kex_parse_kexinit:
    debug2: kex_parse_kexinit: first_kex_follows 0
    debug2: kex_parse_kexinit: reserved 0
    debug2: kex_parse_kexinit: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
    debug2: kex_parse_kexinit: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256
    debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
    debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
    debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-256-96,hmac-sha2-512,hmac-sha2-512-96,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
    debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-256-96,hmac-sha2-512,hmac-sha2-512-96,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
    debug2: kex_parse_kexinit: none,zlib@openssh.com
    debug2: kex_parse_kexinit: none,zlib@openssh.com
    debug2: kex_parse_kexinit:
    debug2: kex_parse_kexinit:
    debug2: kex_parse_kexinit: first_kex_follows 0
    debug2: kex_parse_kexinit: reserved 0
    debug2: mac_setup: found hmac-md5
    debug1: kex: server->client aes128-ctr hmac-md5 none
    debug2: mac_setup: found hmac-md5
    debug1: kex: client->server aes128-ctr hmac-md5 none
    debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
    debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
    debug2: dh_gen_key: priv key bits set: 124/256
    debug2: bits set: 526/1024
    debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
    debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
    debug1: Server host key: RSA 4c:94:a7:2d:b9:b3:b5:a2:1d:37:89:c8:84:d9:ed:bf
    debug3: put_host_port: [127.0.0.1]:54973
    debug3: put_host_port: [localhost]:54973
    debug3: load_hostkeys: loading entries for host "[localhost]:54973" from file "/dev/null"
    debug3: load_hostkeys: loaded 0 keys
    debug1: checking without port identifier
    debug3: load_hostkeys: loading entries for host "localhost" from file "/dev/null"
    debug3: load_hostkeys: loaded 0 keys
    Warning: Permanently added '[localhost]:54973' (RSA) to the list of known hosts.
    debug2: bits set: 519/1024
    debug1: ssh_rsa_verify: signature correct
    debug2: kex_derive_keys
    debug2: set_newkeys: mode 1
    debug1: SSH2_MSG_NEWKEYS sent
    debug1: expecting SSH2_MSG_NEWKEYS
    debug2: set_newkeys: mode 0
    debug1: SSH2_MSG_NEWKEYS received
    debug1: Roaming not allowed by server
    debug1: SSH2_MSG_SERVICE_REQUEST sent
    debug2: service_accept: ssh-userauth
    debug1: SSH2_MSG_SERVICE_ACCEPT received
    debug2: key: /Users/inkel/.docker/machine/machines/dev/id_rsa (0x7feeca600150), explicit
    debug1: Authentications that can continue: publickey,password,keyboard-interactive
    debug3: start over, passed a different list publickey,password,keyboard-interactive
    debug3: preferred publickey,keyboard-interactive
    debug3: authmethod_lookup publickey
    debug3: remaining preferred: keyboard-interactive
    debug3: authmethod_is_enabled publickey
    debug1: Next authentication method: publickey
    debug1: Offering RSA public key: /Users/inkel/.docker/machine/machines/dev/id_rsa
    debug3: send_pubkey_test
    debug2: we sent a publickey packet, wait for reply
    debug1: Server accepts key: pkalg ssh-rsa blen 279
    debug2: input_userauth_pk_ok: fp 8b:f1:19:49:34:18:61:8c:ba:cd:a5:65:99:aa:ce:ea
    debug3: sign_and_send_pubkey: RSA 8b:f1:19:49:34:18:61:8c:ba:cd:a5:65:99:aa:ce:ea
    debug1: read PEM private key done: type RSA
    debug1: Authentication succeeded (publickey).
    Authenticated to localhost ([127.0.0.1]:54973).
    debug1: setting up multiplex master socket
    debug3: muxserver_listen: temporary control path /Users/inkel/.ssh/master-docker@localhost:54973.bGmwLqk9AzPAnw76
    debug2: fd 6 setting O_NONBLOCK
    debug3: fd 6 is O_NONBLOCK
    debug3: fd 6 is O_NONBLOCK
    debug1: channel 0: new [/Users/inkel/.ssh/master-docker@localhost:54973]
    debug3: muxserver_listen: mux listener channel 0 fd 6
    debug1: control_persist_detach: backgrounding master process
    debug2: control_persist_detach: background process is 15146
    debug2: fd 6 setting O_NONBLOCK
    debug1: forking to background
    debug1: Entering interactive session.
    debug2: set_control_persist_exit_time: schedule exit in 300 seconds
    debug1: multiplexing control connection
    debug3: fd 7 is O_NONBLOCK
    debug3: fd 7 is O_NONBLOCK
    debug1: channel 1: new [mux-control]
    debug3: channel_post_mux_listener: new mux channel 1 fd 7
    debug3: mux_master_read_cb: channel 1: hello sent
    debug2: set_control_persist_exit_time: cancel scheduled exit
    debug3: mux_master_read_cb: channel 1 packet type 0x00000001 len 4
    debug2: process_mux_master_hello: channel 1 slave version 4
    debug2: mux_client_hello_exchange: master version 4
    debug3: mux_client_forwards: request forwardings: 0 local, 0 remote
    debug3: mux_client_request_session: entering
    debug3: mux_client_request_alive: entering
    debug3: mux_master_read_cb: channel 1 packet type 0x10000004 len 4
    debug2: process_mux_alive_check: channel 1: alive check
    debug3: mux_client_request_alive: done pid = 15147
    debug3: mux_client_request_session: session request sent
    debug3: mux_master_read_cb: channel 1 packet type 0x10000002 len 92
    debug2: process_mux_new_session: channel 1: request tty 0, X 1, agent 0, subsys 0, term "xterm", cmd "sudo /etc/init.d/docker restart", env 1
    debug3: process_mux_new_session: got fds stdin 8, stdout 9, stderr 10
    debug1: channel 2: new [client-session]
    debug2: process_mux_new_session: channel_new: 2 linked to control channel 1
    debug2: channel 2: send open
    debug2: callback start
    debug2: client_session2_setup: id 2
    debug2: fd 5 setting TCP_NODELAY
    debug3: packet_set_tos: set IP_TOS 0x08
    debug1: Sending environment.
    debug1: Sending env LANG = en_US.UTF-8
    debug2: channel 2: request env confirm 0
    debug1: Sending command: sudo /etc/init.d/docker restart
    debug2: channel 2: request exec confirm 1
    debug3: mux_session_confirm: sending success reply
    debug2: callback done
    debug2: channel 2: open confirm rwindow 0 rmax 32768
    debug1: mux_client_request_session: master session id: 2
    debug2: channel 2: rcvd adjust 2097152
    debug2: channel_input_status_confirm: type 99 id 2
    debug2: exec request accepted on channel 2
    debug1: client_input_channel_req: channel 2 rtype exit-status reply 0
    debug3: mux_exit_message: channel 2: exit message, evitval 0
    debug1: client_input_channel_req: channel 2 rtype eow@openssh.com reply 0
    debug2: channel 2: rcvd eow
    debug2: channel 2: close_read
    debug2: channel 2: input open -> closed
    debug2: channel 2: rcvd eof
    debug2: channel 2: output open -> drain
    debug2: channel 2: obuf empty
    debug2: channel 2: close_write
    debug2: channel 2: output drain -> closed
    debug2: channel 2: rcvd close
    debug3: channel 2: will not send data after close
    debug2: channel 2: send close
    debug2: channel 2: is dead
    debug2: channel 2: gc: notify user
    debug3: mux_master_session_cleanup_cb: entering for channel 2
    debug2: channel 1: rcvd close
    debug2: channel 1: output open -> drain
    debug2: channel 1: close_read
    debug2: channel 1: input open -> closed
    debug2: channel 2: gc: user detached
    debug2: channel 2: is dead
    debug2: channel 2: garbage collecting
    debug1: channel 2: free: client-session, nchannels 3
    debug3: channel 2: status: The following connections are open:
      #2 client-session (t4 r0 i3/0 o3/0 fd -1/-1 cc -1)
    
    debug2: channel 1: obuf empty
    debug2: channel 1: close_write
    debug2: channel 1: output drain -> closed
    debug2: channel 1: is dead (local)
    debug2: channel 1: gc: notify user
    debug3: mux_master_control_cleanup_cb: entering for channel 1
    debug2: channel 1: gc: user detached
    debug2: channel 1: is dead (local)
    debug2: channel 1: garbage collecting
    debug1: channel 1: free: mux-control, nchannels 2
    debug3: channel 1: status: The following connections are open:
    
    debug2: set_control_persist_exit_time: schedule exit in 300 seconds
    debug3: mux_client_read_packet: read header failed: Broken pipe
    debug2: Received exit status from master 0
    
14. inkel commented on Jul 15, 2015

    

    inkel

    on Jul 15, 2015

    More actions

    Mmm... after a few minutes, the following appeared on my screen:

    debug1: ControlPersist timeout expired
    debug1: channel 0: free: /Users/inkel/.ssh/master-docker@localhost:54973, nchannels 1
    debug3: channel 0: status: The following connections are open:
    
    debug3: fd 0 is not O_NONBLOCK
    debug3: fd 1 is not O_NONBLOCK
    Transferred: sent 3104, received 2480 bytes, in 301.1 seconds
    Bytes per second: sent 10.3, received 8.2
    debug1: Exit status -1
    
15. nathanleclaire commented on Jul 15, 2015

    

    nathanleclaire

    on Jul 15, 2015

    Contributor

    More actions

    @inkel Odd. Do you have a VPN or any unusual setup as far as your network goes that you can think of?
16. 22 remaining items

    Load more

17. dracan commented on Oct 9, 2015

    

    dracan

    on Oct 9, 2015

    More actions

    I get this issue after my host machine has gone to sleep, and I come back to it later. I end up deleting everything from my c:\users\dan.docker\machine folder (other than the cache), then recreate it all again. It's quicker doing that than rebooting the host machine! ;)
18. dracan commented on Oct 9, 2015

    

    dracan

    on Oct 9, 2015

    More actions

    Oh, and I also have to kill some processes before doing that. I tend to have 3 'VBoxHeadless.exe' processes, 3 'VBoxNetDHCP.exe', and a 'VirtualBox Interface'.
19. mrumpf commented on Oct 14, 2015

    

    mrumpf

    on Oct 14, 2015

    More actions

    I woke up my notebook and deleted a docker-machine instance with name "dev".
    I ran the following commands to create two new virtual machines:

    $ docker-machine create \
        --driver virtualbox \
        --engine-env HTTP_PROXY=http://10.206.246.20:8080 \
        --engine-env HTTPS_PROXY=http://10.206.246.20:8080 \
        --virtualbox-hostonly-cidr "169.254.0.20/16" \
        registry  
    $ eval $(docker-machine env registry --shell=bash)
    $ docker ps
    CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
    $ docker run -d -p 5000:5000 --restart=always --name registry registry
    $ docker ps
    CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS                    NAMES
    7001a595cc28        registry            "docker-registry"   27 seconds ago      Up 24 seconds       0.0.0.0:5000->5000/tcp   registry
    $ docker-machine ip registry
    169.254.0.100
    $ docker-machine create \
        --driver virtualbox \
        --engine-env HTTP_PROXY=http://10.206.246.20:8080 \
        --engine-env HTTPS_PROXY=http://10.206.246.20:8080 \
        --engine-insecure-registry "$(docker-machine env registry):5000" \
        --virtualbox-hostonly-cidr "169.254.0.20/16" \
        dev
    $ docker-machine ip dev
    169.254.0.101
    $ eval $(docker-machine env dev --shell=bash)
    

    The docker-machine env dev --shell=bash hangs for some minutes and dumps the following to stderr:

    C:\Program Files\ConEmu>docker-machine -D env dev --shell=bash > c:\tmp\docker-machine_env.log
    Maximum number of retries (60) exceeded
    

    The stdout log output can be found in this Gist.

    The strange thing is that the correct output is dumped after the maximum number of retries is reached, see the end of the log.

    This is the timed stdout output:

    $ time docker-machine env dev --shell=bash
    Maximum number of retries (60) exceeded
    export DOCKER_TLS_VERIFY="1"
    export DOCKER_HOST="tcp://169.254.0.101:2376"
    export DOCKER_CERT_PATH="C:\Users\mrumpf\.docker\machine\machines\dev"
    export DOCKER_MACHINE_NAME="dev"
    # Run this command to configure your shell:
    # eval "$(C:\cygwin64\home\mrumpf\bin\docker-machine.exe env dev)"
    
    real    4m8.802s
    user    0m0.015s
    sys     0m0.078s
    

    And it keeps getting stranger...
    The same command for the first virtual machine is executed without any issue:

    $ time docker-machine env registry --shell=bash
    export DOCKER_TLS_VERIFY="1"
    export DOCKER_HOST="tcp://169.254.0.100:2376"
    export DOCKER_CERT_PATH="C:\Users\mrumpf\.docker\machine\machines\registry"
    export DOCKER_MACHINE_NAME="registry"
    # Run this command to configure your shell:
    # eval "$(C:\cygwin64\home\mrumpf\bin\docker-machine.exe env registry)"
    
    real    0m2.129s
    user    0m0.000s
    sys     0m0.093s
    
20. dgageot

    added

    driver/virtualbox

    on Oct 14, 2015
21. nathanleclaire commented on Oct 14, 2015

    

    nathanleclaire

    on Oct 14, 2015

    Contributor

    More actions

    @mrumpf If you add a trailing slash to the *PROXY options e.g. --engine-env HTTP_PROXY=http://10.206.246.20:8080/, does it succeed or continue failing?
22. remram44 commented on Oct 15, 2015

    

    remram44

    on Oct 15, 2015

    More actions

    I'm getting this right now on Debian 8.2.
23. jrep commented on Oct 29, 2015

    

    jrep

    on Oct 29, 2015

    More actions

    Add Jupiter Junos Pulse to the list of VPN software that seems to cause this problem.

    It strikes me as likely that any VPN would do this, since they all have the same fundamental definition to override "ordinary" routing.

    Update: by using a "split connection" (an option that my installation allows. by connecting to a different VPN server), I seem to avoid this.
24. remram44 commented on Oct 29, 2015

    

    remram44

    on Oct 29, 2015

    More actions

    Wait what? I'm not using any kind of VPN or proxy...
25. jrep commented on Oct 29, 2015

    

    jrep

    on Oct 29, 2015

    More actions

    VPN is just one of many things that mess with network configuration. More generally, any change in network connections might (and that includes some things you might not think of as "networking"):

    • VPN enable/disable
    • Wired LAN connect/disconnect
    • WiFi enable / disable / zone change (like carrying a laptop to another area of the building)
    • Bluetooth device connect/disconnect (because of Bluetooth PAN)
    • even USB device connect/disconnect/on/off

    The VPN case is noteworthy because it may offer the "split" option, which may avoid flummoxing the Docker connection from this cause. But split VPN isn't going to prevent all those other things from messing things up.
26. remram44 commented on Oct 29, 2015

    

    remram44

    on Oct 29, 2015

    More actions

    I'm using an old-fashioned RJ45 cable...

    I don't understand why docker-machine is that sensitive to these events.
27. dgageot commented on Oct 29, 2015

    

    dgageot

    on Oct 29, 2015

    Contributor

    More actions

    I tried rj45 plug/unplug, wifi switching. No issue on my side.
28. jrep commented on Nov 3, 2015

    

    jrep

    on Nov 3, 2015

    More actions

    Could there be a need for some processing time? I keep running into this with a shell function that does:

    docker-machine stop "${VM}"
    docker-machine start "${VM}"
    docker-machine ssh "${VM}" sudo /etc/init.docker restart
    eval $(docker-machine env "${VM}"
    

    When I do these commands one at a time, with my fingers, it goes fine. But when run from the script, the same command sequence works once, than hangs (or takes several minutes).

    Connecting to @mrumpf , this VM is the second in my docker-machine ls list.
29. nathanleclaire commented on Nov 3, 2015

    

    nathanleclaire

    on Nov 3, 2015

    Contributor

    More actions

    @jrep Ah, in that case, yes, you definitely need to wait a brief interval for the daemon to start up and begin accepting requests. It's why Machine has code to wait for Docker in between our daemon restarts during provisioning. Arguably, we should check for that on start as well.
30. nathanleclaire commented on Nov 5, 2015

    

    nathanleclaire

    on Nov 5, 2015

    Contributor

    More actions

    This issue is very long and contains a lot of digressions. If someone continues to encounter similar ones please open a new one at https://github.com/docker/machine/issues/new with detailed information including:

    • Which OS you are on
    • Anything atypical in networking configuration (VPN, proxy, SSH configuration settings, etc.),
    • Output of the misbehaving commands with the --debug flag
    • the VirtualBox logs from ~/.docker/machine/machines/name/name.

    Thanks!

    • N
31. nathanleclaire

    closed this as completedon Nov 5, 2015