Visitar URL original
Hijack is incompatible with use of CloseNotifier · Issue #12845 · moby/moby · GitHub
Skip to content

Hijack is incompatible with use of CloseNotifier #12845

Closed

Description

@delfick

Hi,

I upgraded my docker to 1.6 today (I believe it was on 1.5, but I'm not 100% sure).

I now have a problem where my docker client gives me docker.errors.APIError: 500 Server Error: Internal Server Error ("http: Hijack is incompatible with use of CloseNotifier") when I try to post to /containers/<container>/attach (my client uses dockerpty and docker-py)

It would appear that compose has a similar problem (docker/compose#1275).

If I run an image with the docker cli it does work (for example, docker run -it ubuntu:14.04 /bin/bash)

I'm not entirely sure how to debug this issue.

Thanks

Stephen.

Activity

  1. LK4D4 commented on Apr 28, 2015

    @LK4D4
    Contributor

    Hmm, this is pretty weird, compose running into this on build and that's the place where we use CloseNotifier.

  2. LK4D4 commented on Apr 28, 2015

    @LK4D4
    Contributor

    ping @pwaller
    Seems like attach just crashing build if client supports CloseNotifier.

  3. pwaller commented on Apr 28, 2015

    @pwaller
    Contributor

    I don't understand how this is happening. There's only one call to CloseNotifier, and it's in postBuild. How are we ending up calling Hijack thereafter on the same w?

    https://github.com/docker/docker/search?utf8=%E2%9C%93&q=CloseNotifier&type=Code

  4. LK4D4 commented on Apr 28, 2015

    @LK4D4
    Contributor

    @pwaller Dunno :/ Maybe connection is used for two requests?

  5. pwaller commented on Apr 28, 2015

    @pwaller
    Contributor

    @LK4D4 could it be a go bug then? That is really weird. It would be useful to see the exact sequence of events and connections involved. I'm not likely to spend a lot of time delving into this, though it has piqued my curiosity a bit.

  6. LK4D4 commented on Apr 28, 2015

    @LK4D4
    Contributor

    @pwaller I wonder what about keep-alive, so you first calling Build, then Attach and hijack in attach failing.

  7. LK4D4 commented on Apr 28, 2015

    @LK4D4
    Contributor

    @delfick Do you build something before attach?

  8. pwaller commented on Apr 28, 2015

    @pwaller
    Contributor

    @LK4D4 I guess it could be keep-alive, but I would expect the effects of the CloseNotifier to have finished by the time the connection is ready to be reused for the attach. (hence my suggestion that it could be a bug).

  9. LK4D4 commented on Apr 28, 2015

    @LK4D4
    Contributor
  10. pwaller commented on Apr 28, 2015

    @pwaller
    Contributor

    @LK4D4 nice find, good to see it's being fixed in general.

    However, I still don't understand the sequence of events that is causing the problem as reported here. Do we think someone fires off a build, and at some point later tries to attach, and they're still using the same socket to communicate to the daemon? I'm surprised that this triggers the issue. I guess the workaround is to tell clients to kill their keep-alive after a build or before an attach :(

  11. LK4D4 commented on Apr 28, 2015

    @LK4D4
    Contributor

    @pwaller Yeah, seems like many users of api can trigger this with Keep-Alive. I already suggested them to reopen connections in docker/compose#1275

  12. delfick commented on Apr 28, 2015

    @delfick
    Author

    I agree, removing the build step makes it work (my docker client reads yaml that embeds the dockerfile and so it always rebuilds the image before running)

    I'll put in code that recreates the connection for now. Good to see there's an actual fix in the pipeline.

  13. mko-x commented on May 10, 2015

    @mko-x

    Just as info: +1

    Stacktrace may help a little:

    Traceback (most recent call last):
    File "/compose/build/docker-compose/out00-PYZ.pyz/threading", line 810, in __bootstrap_inner
    File "/compose/build/docker-compose/out00-PYZ.pyz/threading", line 763, in run
    File "/compose/build/docker-compose/out00-PYZ.pyz/compose.cli.multiplexer", line 41, in _enqueue_output
    File "/compose/build/docker-compose/out00-PYZ.pyz/compose.cli.log_printer", line 59, in _make_log_generator
    File "/compose/build/docker-compose/out00-PYZ.pyz/compose.cli.utils", line 77, in split_buffer
    File "/compose/build/docker-compose/out00-PYZ.pyz/docker.client", line 199, in _multiplexed_response_stream_helper
    File "/compose/build/docker-compose/out00-PYZ.pyz/docker.client", line 143, in _get_raw_response_socket
    File "/compose/build/docker-compose/out00-PYZ.pyz/docker.client", line 95, in _raise_for_status
    APIError: 500 Server Error: Internal Server Error ("http: Hijack is incompatible with use of CloseNotifier")

  14. ogavrisevs commented on May 10, 2015

    @ogavrisevs

    +1

    Exception in thread Thread-1:
    Traceback (most recent call last):
    File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
    File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/threading.py", line 484, in run
    self.__target(_self.__args, *_self.__kwargs)
    File "/Library/Python/2.6/site-packages/compose/cli/multiplexer.py", line 41, in _enqueue_output
    for item in generator:
    File "/Library/Python/2.6/site-packages/compose/cli/log_printer.py", line 59, in _make_log_generator
    for line in line_generator:
    File "/Library/Python/2.6/site-packages/compose/cli/utils.py", line 77, in split_buffer
    for data in reader:
    File "/Library/Python/2.6/site-packages/docker/client.py", line 225, in _multiplexed_response_stream_helper
    socket = self._get_raw_response_socket(response)
    File "/Library/Python/2.6/site-packages/docker/client.py", line 167, in _get_raw_response_socket
    self._raise_for_status(response)
    File "/Library/Python/2.6/site-packages/docker/client.py", line 119, in _raise_for_status
    raise errors.APIError(e, response, explanation=explanation)
    APIError: 500 Server Error: Internal Server Error ("http: Hijack is incompatible with use of CloseNotifier")

  15. koliyo commented on May 21, 2015

    @koliyo

    +1

  16. 28 remaining items

  17. pwaller commented on Jul 27, 2015

    @pwaller
    Contributor

    @thaJeztah I personally think this can be closed. I don't think there is anything that can be done docker-side, only "client" side (so, in compose, docker-py, etc etc). There is a workaround available (turn off HTTP connection reuse (aka Keep-Alive)), and it will be fixed when go fixes golang/go#9763 which has been marked 1.6Early. I'll try and create some noise for that when their next release cycle starts.

  18. thaJeztah commented on Jul 27, 2015

    @thaJeztah
    Member

    Thanks @pwaller, I'll close this issue for now, but others feel free to comment here if you think there's something that can be done in Docker itself.

  19. aeikenberry commented on Jul 5, 2016

    @aeikenberry

    I'm getting this error after upgrading today.

    docker info

    Containers: 8
     Running: 2
     Paused: 0
     Stopped: 6
    Images: 282
    Server Version: 1.12.0-rc3
    Storage Driver: aufs
     Root Dir: /var/lib/docker/aufs
     Backing Filesystem: extfs
     Dirs: 320
     Dirperm1 Supported: true
    Logging Driver: json-file
    Cgroup Driver: cgroupfs
    Plugins:
     Volume: local
     Network: host null overlay bridge
    Swarm: inactive
    Runtimes: runc
    Default Runtime: runc
    Security Options: seccomp
    Kernel Version: 4.4.14-moby
    Operating System: Alpine Linux v3.4
    OSType: linux
    Architecture: x86_64
    CPUs: 4
    Total Memory: 1.954 GiB
    Name: moby
    ID: JRAU:RVZS:OEOQ:JMBF:PRR6:7TPD:UG5U:TRR6:4T2W:AGWF:5TFV:SLD2
    Docker Root Dir: /var/lib/docker
    Debug Mode (client): false
    Debug Mode (server): true
     File Descriptors: 29
     Goroutines: 47
     System Time: 2016-07-05T20:24:15.611271815Z
     EventsListeners: 1
    No Proxy: *.local, 169.254/16
    Username: aeikenberry
    Registry: https://index.docker.io/v1/
    Experimental: true
    Insecure Registries:
     127.0.0.0/8
    

    For example:
    docker-compose run django python manage.py migrate

    http: Hijack is incompatible with use of CloseNotifier in same ServeHTTP call
    http: Hijack is incompatible with use of CloseNotifier in same ServeHTTP call
    
  20. cpuguy83 commented on Jul 6, 2016

    @cpuguy83
    Member

    @aeikenberry Thanks for the report, this is a bug with docker4mac and is pretty much being tracked in docker/compose#3685

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions