Overwhelming error output size for large expressions (f.e. using dhall-to-cabal)

Hi, the output of an error includes the original resolved expression. For large expressions like the generated by dhall-to-cabal the size is almost impossible to handle: it has 327.000 lines and 12.469.103 bytes

It seems to me that it is a dhall issue ad not a dhall-to-cabal one, sorry if i am wrong

This happens to us as well, with dhall-kubernetes

Maybe it'd be nice to have a new default minimal error level that cuts away the types over a certain level of nesting, and/or leaves out some things.

@f-f

@f-f Something similar was suggested in #532 (comment). Anothe option could be :

And what about include -error-* flags to enable/disable each of possible elements of the output: type diff(?), explanation, prose diff, and original expression?
It adds complexity to the command line interface, but it seems the possible outputs in a error are somewhat fixed.
Or at least show the error at the beginning...

I would suggest just omitting the context from the default output since the default output should be the minimal and concise version. I just underestimated how large the context would get when I first designed the error messages

Yet another possibility would be capping the output of each displayed expression. There are several ways that we can limit and/or reduce the size of an expression such as:

number of lines (i.e. cutting away the interior and replacing with …)
depth (i.e. not showing more than N levels deep for things like nested records)
common sub-expression elimination using let (i.e. computing the semantic hash of each subexpression and factoring out sub-expressions with identical contents)

Truncate the displayed source code when it exceeds 6 lines

Pull request up here: #589

Let me know if that improvement is enough for you or if you also need me to reduce the size of the displayed context, too.

(I've deleted a comment with a false negative)
I've test the output for a simple dhall input and it is truncated:

C:\dhall-haskell>type test.dhall
{a=1
,b=2
,c=3
,d=4
,f=5
,g=6
,h=7
}.x

C:\dhall-haskell>dhall < test.dhall

 Use "dhall --explain" for detailed errors [0m

 Error: Missing record field

{a=1
,b=2
,c=3
✂��┈��┈��┈��┈��┈��┈��┈��✂��┈��┈��┈��┈��┈��┈��┈��✂��┈��┈��┈��┈��┈��┈��┈��✂��┈��┈�
�┈��┈��┈��┈��┈��✂��┈��┈��┈��┈��┈��┈��┈��✂��┈��┈��┈��┈��┈��┈��┈��✂��┈��┈��┈��┈��┈
��┈��┈��✂��┈��┈��┈��┈��┈��┈��┈��✂��┈��┈��┈��┈��┈��┈��┈��✂��┈��┈��┈��┈��┈��┈��┈��

,g=6
,h=7
}.x

(stdin):1:1

(the weird chars are displayed in a win dos console)

However, testing it with a dhall-to-cabal file (modified to trigger a Missing record field) still prints the large output:

C:\dhall-to-cabal>dhall < dhall-to-cabal.dhall > output.txt 2>&1
C:\dhall-to-cabal>forfiles /m output.txt /c "cmd /c echo @fsize"

14148813

Otoh, not sure if the truncated output could be useful cause the error location probably will be out of the range printed. Could we let users have more fine grained control over the elements printed and/or its size?

@jneira

@jneira: I'm trying to avoid adding lots of flags to tweak the error output because the users that will need those flags the most are new users and they won't even know the flags exist. Also, in general, I feel like the interpreter should do the right thing out-of-the-box and have sensible defaults

Which part of the dhall-to-cabal output is the largest part at the moment? I can apply similar truncations to other parts of the output like the context or the expressions displayed in --explain output.

I can also have the "snip" line respect the --ascii flag so that it works in consoles that don't support UTF8

Actually, it turns out that it's not easy to parametrize the error message on the command line flag because it's part of a type class instance, so I'll just have to resign to always using ASCII, instead of fancy scissors 😢

I'm going to also apply the same "snip" to the expressions in the context and the --explain output

@jneira

@jneira: Alright, try the pull request now. It should now also snip type-checking contexts and the code inserts for detailed errors. For example, this code:

    let y = < Foo0 = {}
            | Foo1 : {}
            | Foo2 : {}
            | Foo3 : {}
            | Foo4 : {}
            | Foo5 : {}
            | Foo6 : {}
            | Foo7 : {}
            | Foo8 : {}
            | Foo9 : {}
            >

in  let x = [ 1
            , 1
            , 1
            , 1
            , 1
            , 1
            , 1
            , 1
            , 1
            , 1
            , 1
            , 1
            , 1
            , 1
            , 1
            ] # [ { a = 1
                  , b = 2
                  , c = 3
                  , d = 4
                  , e = 5
                  , f = 6
                  , g = 7
                  } ]

in   x

... produces this error message:

$ dhall --explain <<< './test.dhall'


↳ ./test.dhall

y : < Foo0 :
        Type
    | Foo1 :
    ========
    | Foo9 :
        {}
    >


Error: You can only append ❰List❱s with matching element types

- Natural
+ { … : … }

Explanation: You can append two ❰List❱s using the ❰#❱ operator, like this:


    ┌────────────────────┐
    │ [1, 2, 3] # [4, 5] │
    └────────────────────┘


... but you cannot append two ❰List❱s if they have different element types.
For example, the following expression is not valid:


       These elements have type ❰Natural❱
       ⇩
    ┌───────────────────────────┐
    │ [1, 2, 3] # [True, False] │  Invalid: the element types don't match
    └───────────────────────────┘
                  ⇧
                  These elements have type ❰Bool❱


────────────────────────────────────────────────────────────────────────────────

You tried to append a ❰List❱ thas has elements of type:

↳ Natural


... with another ❰List❱ that has elements of type:

↳ { a :
      Natural
  , b :
  ===========
  , g :
      Natural
  }


... and those two types do not match

────────────────────────────────────────────────────────────────────────────────

            [ 1
            , 1
            , 1
            =============
                  , g = 7
                  } ]


/Users/gabriel/proj/dhall/test.dhall:13:13

With dhall-to-cabal:

C:\dhall-to-etlas>dhall < dhall-to-cabal.dhall

 
Use "dhall --explain" for detailed errors

prelude : { SPDX :
              { and :
                    ∀ ( a
          =========================================================
              → ∀(invertVersionRange : VersionRange → VersionRange)
              → VersionRange
          }
v : ∀(str : Text) → ∀(Version : Type) → ∀(v : Text → Version) → Version
anyVersion :   ∀(VersionRange : Type)
             → ∀(anyVersion : VersionRange)
             → ∀(noVersion : VersionRange)
             =========================================================================
             → ∀(differenceVersionRanges : VersionRange → VersionRange → VersionRange)
             → ∀(invertVersionRange : VersionRange → VersionRange)
             → VersionRange
package :   ∀(package : Text)
          → ∀ ( version-range
              :   ∀(VersionRange : Type)
          ==============================
            , package :
                Text
            }
majorVersions :   ∀(package : Text)
                → ∀(versions : List (∀(Version : Type) → ∀(v : Text → Version) → Version))
                → { bounds :
                ==========================================================================
                  , package :
                      Text
                  }
deps : { Cabal :
           { bounds :
                 ∀(VersionRange : Type)
       ================================
               Text
           }
       }
warning-options : List Text

Error: Missing record field

                      deps.basex
                    
(stdin):264:23

The output is truncated now, thanks!

I know that seeing a giant error message is painful, but truncating the output is painful in a different way for small error messages.

For example, dealing with a record with three fields:

{ bin : List { file : Text, name : Text }, name : Text, sourceDirectory : Text }

If I give Dhall the wrong fields and things are small enough, I can see what the problem is:

You tried to invoke the following function:                                     
                                                                                
↳ < Haskell :
      { executables :
          List { executableDirectory : Text, executableName : Text }
  ==================================================================
          Text
      }
  >.JavaScript
                                                                                
... which expects an argument of type or kind:                                  
                                                                                
↳ { bin : List { file : Text, name : Text }, name : Text, sourceDirectory : Text }
                                                                                
... on the following argument:                                                  
                                                                                
↳ { name = "build-browserify", sourceDirectory = "src", version = "1.0.0" }
                                                                                
... which has a different type or kind:                                         
                                                                                
↳ { name : Text, sourceDirectory : Text, version : Text }

If I add the missing field bin (but don't remove the extra field version), and bin is longer than one line, I can't easily see what the problem is:

You tried to invoke the following function:                                     
                                                                                
↳ < Haskell :
      { executables :
          List { executableDirectory : Text, executableName : Text }
  ==================================================================
          Text
      }
  >.JavaScript
                                                                                
... which expects an argument of type or kind:                                  
                                                                                
↳ { bin : List { file : Text, name : Text }, name : Text, sourceDirectory : Text }
                                                                                
... on the following argument:                                                  
                                                                                
↳ { bin =
      [ { file = "src/index.js", name = "build-browserify" } ]
  , name =
  ============================================================
  , version =
      "1.0.0"
  }
                                                                                
... which has a different type or kind:                                         
                                                                                
↳ { bin :
      List { file : Text, name : Text }
  , name :
  =====================================
  , version :
      Text
  }

It works out in this case because I put version as the extra field. If path is the extra field, I get a really cryptic error:

You tried to invoke the following function:                                     
                                                                                
↳ < Haskell :
      { executables :
          List { executableDirectory : Text, executableName : Text }
  ==================================================================
          Text
      }
  >.JavaScript
                                                                                
... which expects an argument of type or kind:                                  
                                                                                
↳ { bin : List { file : Text, name : Text }, name : Text, sourceDirectory : Text }
                                                                                
... on the following argument:                                                  
                                                                                
↳ { bin =
      [ { file = "src/index.js", name = "build-browserify" } ]
  , name =
  ============================================================
  , sourceDirectory =
      "src"
  }
                                                                                
... which has a different type or kind:                                         
                                                                                
↳ { bin :
      List { file : Text, name : Text }
  , name :
  =====================================
  , sourceDirectory :
      Text
  }

All of the context I need to figure out the problem has been truncated away.

Showing everything is painful when the context is huge. Choosing an arbitrary number of lines to truncate out is impossible when the context is slightly over that number. Is there some compromise we can come to? I know we're focused on making large config files easy to deal with, but we're going to turn people away before they gain momentum if a record of three fields gives this sort of output.

Maybe the solution here isn't to focus on truncating? The real issue is that the record has the wrong fields. Maybe we can provide a better error message specific for records?

In the last example, if you don't use --explain, you get this output:

Error: Wrong type of function argument

{ + path : …
, …
}

      Package.JavaScript
      { bin =
          [ { file = "src/index.js", name = "build-browserify" } ]
      ============================================================
          "src"
      }

That

{ + path : …
, …
}

message is a bit hard to parse. Does it mean that I need to add path? If so, what's its type? Does it mean I need to remove path? It's especially hard when it's a nested record of who knows how deep:

Error: Wrong type of function argument

{ + path : …
,   bin : …
          { name : - Text
                   + { … : … }
          , …
          }
, …
}

      Package.JavaScript
      { bin =
          [ { file =
      ==================
          "src"
      }

What's the issue here? How should I go about solving it? Should I throw some empty record in there and see what happens? Is there being a record the problem?

The current error message gives you clues about the problem, but it doesn't give you solutions to the problem. If we could say (in prose) that field foo is not in the record, so add it with the type Foo, that would be a step up. If we could say that bar is the right field, but it's got the wrong type and this is what the type should be, that would be a step up.

I recognize that we can't always know what the solution is. The error message could still be a heuristic for fixing the problem. If we say, "it's likely that doing X will solve the problem," that seems better than saying, "here are the things that are wrong, good luck!"

I also recognize that --explain provides a ton of information in the general case about the problem. That's super useful if you don't understand the general issue. Once you do understand general issue, you need a bit more help to solve the specific problem.

@joneshf

Totally agree with @joneshf that error messages should be improved, although there are a delicate balance between conciseness and info, help for different type of users and so on...
I think truncation should be replaced with the "path" to the offending piece of code, using the lines of code of each nested definition (if present). I am afraid that, in practice, i've not found so useful the truncated version.
Maybe it worths a new brand issue about the topic (error reporting desing)?

Yeah, if you open a new issue I can address this

Navigation Menu

Repository navigation

Overwhelming error output size for large expressions (f.e. using dhall-to-cabal) #573

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Overwhelming error output size for large expressions (f.e. using dhall-to-cabal) #573

Description

Activity

f-f commented on Sep 12, 2018

jneira commented on Sep 12, 2018

Gabriella439 commented on Sep 12, 2018

Gabriella439 commented on Sep 12, 2018

Gabriella439 commented on Sep 18, 2018

jneira commented on Sep 18, 2018

jneira commented on Sep 18, 2018

Gabriella439 commented on Sep 18, 2018

Gabriella439 commented on Sep 20, 2018

Gabriella439 commented on Sep 20, 2018

jneira commented on Sep 20, 2018

joneshf commented on Mar 19, 2019

joneshf commented on Mar 19, 2019

jneira commented on Mar 19, 2019

Gabriella439 commented on Mar 19, 2019

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions