Repository navigation
Tags
Tags
Add literal twins of Bash loop values (#242) * Add literal twins of Bash loop values Add BashParser.TryProjectLiteralTwins. For each occurrence whose changeable argument words have a proved finite set of values, the projection writes one literal twin for each combination of values. A twin is the full source with each such word written as one literal word. The parser parses it again, so the twin occurrence has the facts of the typed literal: command words, path facts, and expansion facts. An occurrence gets no twins when it is incomplete, when a verb word is not static, when a word has no complete finite value set, when an unquoted value can split or glob, when a redirect depends on a value, or when the combinations pass the limit (32 for one occurrence, 128 in total). An unquoted word uses its authored values only under FreshNonInteractiveNoStartup, and only when each value is nonempty and has no space, tab, newline, glob character, or backslash. The rules use general shell facts only. A twin grants no authority. Parse output does not change. Prepare 0.4.0-beta.22. * Close review round 1 gaps in literal twins Give no twins for text that the lexer does not model as Bash does: - a source with a backslash-newline or a carriage return; - an occurrence with `=~` or `:~` in a word; - an occurrence in a source with `~` or `HOME` when no live launch HOME proves the home directory; - an unquoted value with a parenthesis, which an extglob pattern needs. Count every twin parse, also a failed one, against one budget for a call: 128 parses and 32,768 source characters. An occurrence whose parses do not fit gets no twins and costs no parse. Run the cheap checks for all occurrences first. Add a Linux test that runs each source and each twin in GNU Bash and compares the argv. Document the consumer warnings. Move the release to 0.4.0-beta.23, after the beta.22 lexer fix. * Parse literal twins as single commands and guard list tildes Check `=~` and `:~` in the full source, as the backslash-newline and carriage-return guards do. Bash expands a tilde in a loop list item such as `for f in if=~/x`, and the parser keeps it as text. Parse each twin text as one command, not as the full source. The twin text is the authored text from the first to the last word of the command. The parser parses it in the exact directory of the command, with the launch facts that are live there. A twin parse then costs about its own length. Compute the length of every twin text before any build, and check the budget first: 128 twin parses and 2,048 + 8 x (source length) characters. The projection now costs about one Parse. On an 80 KB source with 8,000 commands it takes about as long as Parse (before: 23 s and 80 GB). A command gets no twins when its directory is not exact, when it has an assignment prefix, or when it has a heredoc. BashLiteralTwin gets a WorkingDirectory property. * Give no twins under an unknown state or an unproved home directory Under BashInitialStateMode.Unknown, the parser can report an exact value for a word that Bash expands: `-o"$n"` gives `-o$n`, but Bash passes `-oa`. The projection now gives no twins in that mode. Parse is not changed here. A cd without an operand goes to $HOME. Without a live launch HOME, the parser takes that directory from HomeDirectory, but Bash uses its real HOME. The projection now parses the source once more with another HomeDirectory. A command whose exact directory changes gets no twins. The check does not read the cd forms, so it also covers `cd --`, `builtin cd`, `command cd`, and `if cd; then`. Add the inputs as tests and to the real-Bash comparison, which now also compares the working directory. Fix the stale parser comment. * Prepare 0.4.0-beta.23 on top of the beta.22 lexer fix Add the 0.4.0-beta.23 release notes above 0.4.0-beta.22, and set the README version line to 0.4.0-beta.23. Move the beta.22 plan section to Completed. Since 0.4.0-beta.22, Parse gives Unknown or the Bash value for a loop list item with `=~` or `:~`, never the text. Update the real-Bash test to that rule. Keep the line-text guards as defense in depth and say so in the code, the SPEC, and the consumer guide. Add a carriage return in single quotes as a guard test, because a carriage return outside quotes is now unparseable.
Fix Bash parsing that hides commands: continuations, CR, and word # (#… …243) * Remove Bash line continuations before reading an expansion Bash removes a backslash-newline pair before it splits tokens, except in single quotes, $'...', a comment, and a heredoc body. The lexer read "$\<newline>(cmd)" as literal text, so cmd ran with no occurrence. Every lookahead that reads the second character of a multi-character construct now skips line continuations through one helper (BashLineContinuation): $, $(, ${, $name, $', operators, here-doc operators, and descriptor targets. Later readers of authored text (redirect operators, word-expansion facts, command words, option spelling) use the same rule. A split $(( marker, a continuation in an expanding heredoc body or delimiter, and $'...' with a continuation fail closed. * Add tests that kill the surviving continuation mutants * Pin the substitution scan with a quoted inner substitution * Remove a dead arithmetic-marker guard that a mutant exposed * Prepare ShellSyntaxTree 0.4.0-beta.22 * Fail closed on a carriage return in Bash code Bash reads CR as a word character and backslash + CR as an escaped CR. The lexer read CR as a line end and backslash + CRLF as a continuation, so 'echo a\<CR><LF>touch x' and 'echo a<CR># ; touch x' hid 'touch'. A CR outside quotes, comments, and heredoc bodies now makes the source unparseable, and so does backslash + CR in double quotes. Comments end only at LF, and a heredoc line with a trailing CR is not its delimiter. * Take a Bash inline option value from the decoded word The value argument of '--name=value' came from the authored text after '=' or, with an escape, from the whole word. 'curl --data=@p\.json' gave '--data=@p.json' and 'curl --data=\a""' gave '--data=a'. The value is now the part of the decoded word (or of each proved value) after the option and its '='. Anything else is Unknown. * Pin a CR comment in a substitution and computed inline values * Document the carriage-return and inline-value changes * Record the beta.22 comparison numbers * Use LF in a multi-line Bash test source on every platform * Fix the #243 review: word '#', reserved words, assignment tildes - A '#' starts a comment only when it begins a new word. After a quoted part, $(...), or $((...)) it is word text, as in Bash, so 'echo "a"# ; touch x' shows 'touch'. - Reserved words, keywords, and literal checks read the spelling without line continuations, so 't\<LF>ime touch x' fails closed like 'time'. - In an assignment-shaped word (name= or name+=), a '~' after the first '=' or after ':' expands from HOME, as Bash does outside POSIX mode. '~user', '~+', and a word whose start is not visible give Unknown. A here-string keeps the text. * Add bash-verified tests for the #243 review findings Includes a generated check: a '#' at every point of 13 seeds must match the commands that Bash runs (logged, nothing real runs) or fail closed. The Bash oracle helpers move to a shared BashOracle test class. * Pin the tilde state after an escape, an expansion, or a continuation * Pin a continuation before an unprovable assignment tilde * Read (( and ;; across a line continuation The ((...)) gate and the ;; case terminator compared raw offsets, so '(\<LF>(ls))' parsed as two subshells that run 'ls'. Bash removes the continuation and reads an arithmetic command, which fails closed here. Found by checking that no continuation variant of an unparseable source parses. * Document the #243 review fixes * Record the final beta.22 comparison numbers * Expand an assignment-word tilde only in proved non-POSIX Bash Round 2 of the #243 review. A '~' after '=' or ':' of an assignment-shaped word now expands only in the submitted source (not a decoded sh -c or bash -c child), in FreshNonInteractiveNoStartup mode, with a launch-proved HOME. Everywhere else the value is Unknown. POSIX mode and dash keep the text, and the other modes can inherit POSIX mode. A subscripted name (a[0]=~/x) gives Unknown. A real assignment keeps its leading tilde in every mode. * Document the assignment-word tilde gate * Record the round-2 comparison numbers
Parse bounded arithmetic and loop transfers; unresolve brace words (#230 ) Parse a bounded $((...)) whose variable reads hold proved integers. Fail closed on ((...)), subscripts, assignments, command substitutions, and unproved reads. Parse bounded break, continue, exit, and return, and join loop-transfer states into their loops. Give a word with a brace expansion no exact value and no path. Prepare 0.4.0-beta.18. Refs #227
PreviousNext