Support non-Hackage dependencies

@ezyang

Summary by @ezyang:

Syntactic support for these URLs exists: in the packages field you can specify remote tarballs or local tarballs, and using the source-repository syntax (from Cabal file) you can point to a VCS URL (this parser should also have been reused for cabal.project)
Support for downloading from various places exists. There's support in the old-build path for local/remote tarballs. cabal get -s supports getting things from source repositories.
However, these are not used for anything: the internals need to be plumbed up to use this. Look at old stuff and determine how to port it over.

This should be doable in a day.

This came up on Twitter:

https://twitter.com/darinmorrison/status/527716059889954816

Basically, people want to publish packages that depend on stuff that is not on Hackage. One solution is to upload a forked package (example), but this has its own downsides (Hackage namespace becomes polluted). Implementing something like build-depends: https://repo.git@revision would be a nice alternative. Most other package managers (e.g. sbt) support this.

It should also be possible to prohibit downloading packages not on Hackage via a config file option.

See also #1534.

Halcyon supports declaring sandbox sources with the HALCYON_SANDBOX_SOURCES option.

See Try Haskell for an example of declaring the git HEAD version of mueval as a sandbox source.

I'd like to bump this issue's priority and start to discuss implementation, currently it's stopping us using cabal in company environment because we have multiple packages can't be upload to hackage, and a hackage mirror is an overkill. now we use stack to solve this headache, but it's really not necessary to rely both on hackage and stackage infrastructure.

@winterland1989

Hello @winterland1989. Have you tried the new new-build functionality (http://blog.ezyang.com/2016/05/announcing-cabal-new-build-nix-style-local-builds/). If you have all of the non-Hackagable packages on disk in a build tree, you can write a cabal.project to pick them up, and then cabal new-build will automatically build them together. Does that solve your proximal problem?

@ezyang

Thanks for your reply @ezyang ! I'm aware that there're multiple ways solving local package discovery: cabal sandbox add-source or a cabal.project file both solve this, and i'm aware that cabal.project file support parsing URI as package now. But we need an automatic way to pull/update the remote file into disk without hassle, currently available options are git submodule, hand-role shell and stack.(you really want to automate this process when you got more than 5 local depends).

If cabal.project file is the final solution, i'll dig into it to see what i can do.

@winterland1989

@winterland1989
There is already some code for declaring VCS dependencies in cabal.project files: https://github.com/haskell/cabal/blob/master/cabal-install/Distribution/Client/ProjectConfig.hs#L523

If you want to contribute code to Cabal, I suggest improving the nix-local-build code path, since the plan is to make it the default in the future.

I would like to see something like http://docs.haskellstack.org/en/stable/yaml_configuration/#complex-package-locations-location

Looks like we only need to teach cabal-install how to fetch stuff from the internet:

cabal/cabal-install/Distribution/Client/ProjectConfig.hs

Line 745 in 5423d77

fail $ "TODO: add support for fetching and reading local tarballs, remote "

tar support shouldn't be difficult, as cabal handles it already. git is trickier, as we need to teach cabal-install to call git clone (and darcs).

Question: were to put downloaded stuff, dist-newstyle/downloaded/hash-of-the-source ?

(For now one can use git submodule, effectively the result should be the same)

FYI, every github commit can be downloaded as a zip file, e.g.

https://github.com/haskell/cabal/archive/master.zip
https://github.com/haskell/cabal/archive/59eabde01e7df7ac51602b40d867fe5ece2992c2.zip

FYI, every github commit can be downloaded as a zip file, e.g.

Yes, but that's github's convention, we probably shouldn't rely on that convention. In fact we can safely assume user have git binary installed if they want to use git depends, and we can use process package manage git clone/pull.

A use case for this is during development when another client component has not yet been released to hackage. E.g. ghc-mod for HaRe. This is a temporary state.

Having a temporary config in a cabal.project is a lot more lightweight than adding a git submodule.

If we allow fiddling then it has to be treated like a local dir.

We can ask git if the source has been fiddled with and then fall back to treating it like a local dir if it is the case.

I'm inclined to just not allow specification of a hash if you don't specify a remote URL. Don't really want new-build to be checking out git branches or ignoring my hash...

I'm looking forward to celebrating this issue's second 🎂 in two days.

Happy 🎂!

The provenance-qualified imports proposal (ghc-proposals/ghc-proposals#115) is relevant here.

@bitemyapp

It's a pity @bitemyapp won't get to celebrate schadenfreude this year... ;-)

Closed by #5351.

Exciting! I want to play with this

But is this really fixed? https://github.com/haskell/cabal/blob/master/Cabal/doc/nix-local-build.rst on master still says


    3. [STRIKEOUT:They can specify an ``http``, ``https`` or ``file``
       URL, representing the path to a remote tarball to be downloaded
       and built.] (not implemented yet)

and

[STRIKEOUT:There is also a stanza source-repository-package for specifying packages from an external version control.] (Not implemented.)

That's a documentation bug, I believe.

I will know in a few minutes when cabal has built and I can try the feature :-)
…
Yes, seems to work. Great!

Navigation Menu

Repository navigation

Support non-Hackage dependencies #2189

7 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Support non-Hackage dependencies #2189

Description

Activity

mietek commented on Nov 11, 2014

winterland1989 commented on Jul 11, 2016

ezyang commented on Jul 11, 2016

winterland1989 commented on Jul 11, 2016

23Skidoo commented on Jul 11, 2016

alanz commented on Jul 11, 2016

phadej commented on Jul 13, 2016

phadej commented on Jul 13, 2016

alanz commented on Jul 13, 2016

winterland1989 commented on Jul 13, 2016

alanz commented on Jul 13, 2016

7 remaining items

23Skidoo commented on Sep 8, 2016

ezyang commented on Sep 8, 2016

bitemyapp commented on Oct 28, 2016

bitemyapp commented on Oct 30, 2017

gbaz commented on Mar 24, 2018

hvr commented on Jul 14, 2018

typedrat commented on Jul 15, 2018

nomeata commented on Jul 18, 2018

typedrat commented on Jul 18, 2018

nomeata commented on Jul 18, 2018

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions