Visitar URL original
Support UTF-8 and UTF-16 column units by kaste · Pull Request #2000 · SublimeLinter/SublimeLinter · GitHub
Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 32 additions & 2 deletions docs/linter_attributes.rst
Original file line number Diff line number Diff line change
Expand Up @@ -220,6 +220,33 @@ It filters out deprecation warnings while still keeping other hard errors and
reports them back to the user.


.. _column_unit:

column_unit
-----------
This attribute tells SublimeLinter which unit the linter uses for the reported columns.
Possible values are ``codepoint`` (the default), ``"utf8"`` (bytes), or ``"utf16"``
(16-bit code units).

.. code-block:: python

class Tslint(NodeLinter):
column_unit = "utf16"

class Cppcheck(Linter):
column_unit = "utf8"

Numeric columns still use :ref:`line_col_base`: base subtraction happens in
``split_match``, before source-dependent conversion. A nonnumeric ``col``
capture (e.g. a caret prefix consisting of spaces and tabs) still contributes
its length without base subtraction, now measured in the declared units.
Tabs count as one unit; display-cell, grapheme, and expanded-tab columns are
not supported.

For linters that mix units in their output, yes (!), use the source-aware
:ref:`convert_column` method which is called for each match.


.. _line_col_base:

line_col_base
Expand Down Expand Up @@ -314,9 +341,12 @@ Very often linters don't do that and then it's best to omit capturing ``filename
SublimeLinter will figure out the correct value for you and you don't run into
problems with unnamed buffers or possibly symlinked files.

You can also capture ``end_line`` and ``end_col``, otherwise the :ref:`word<word_re>` beginning at ``col`` will be highlighted. How the numbers are interpreted is defined by :ref:`line_col_base`.
You can also capture ``end_line`` and ``end_col``, otherwise the :ref:`word<word_re>`
beginning at ``col`` will be highlighted. Their base is defined by :ref:`line_col_base`,
and their units by :ref:`column_unit`.

If you can't capture the ``error_type`` directly, you may use ``error`` and ``warning`` to set the type. Alterantively, you fallback to :ref:`default_type`.
If you can't capture the ``error_type`` directly, you may use ``error`` and ``warning``
to set the type. Alternatively, you fallback to :ref:`default_type`.

+------------+-----------------------------------------------------------------+
| error | If this is not empty, the error will be marked |
Expand Down
64 changes: 61 additions & 3 deletions docs/linter_methods.rst
Original file line number Diff line number Diff line change
Expand Up @@ -27,12 +27,70 @@ split_match
-----------
.. code-block:: python

split_match(self, match)
split_match(self, match) -> LintMatch

This method extracts the named capture groups from the :ref:`regex` and
return a tuple of *match*, *line*, *col*, *error*, *warning*, *message*, *near*.
This method extracts named capture groups from the :ref:`regex` into a
``LintMatch`` mapping. It parses line and column numbers and subtracts
:ref:`line_col_base`, but does not convert Unicode column units. Custom
parsers must likewise return zero-based columns in the declared units.

If subclasses need to modify the values returned by the regex,
they should override this method, call ``super().split_match(match)``,
then modify the values and return them.

For compatibility, ``LintMatch`` still supports seven-value tuple iteration
and positional construction: *match*, *line*, *col*, *error*, *warning*,
*message*, *near*. Additional fields do not alter that tuple contract.


.. _convert_column:

convert_column
--------------
.. code-block:: python

convert_column(self, line: int, col: int, m: LintMatch, vv: VirtualView) -> int

Convert a column as reported by the linter to a code-point offset. The default uses
:ref:`column_unit`. Core calls this hook once for each present ``col`` and
``end_col`` in ``process_match``, before either positioning branch clamps
columns. Missing columns do not call the hook; zero columns do.

.. code-block:: python

def convert_column(self, line, col, m, vv):
if m.code.startswith("F"):
return vv.col_from_utf8(line, col)
return super().convert_column(line, col, m, vv)

``vv.col_from_utf8(line, col)`` and ``vv.col_from_utf16(line, col)`` convert
individual zero-based offsets.

For source-specific coordinates, reconstruct the tool's origin in the scalar
hook. For example, xmllint prints a window and its parser stores the caret's
zero-based byte offset as ``col``::

def convert_column(self, line, col, m, vv):
if m.context is not None:
origin = max(vv.select_line(line).find(m.context), 0)
window = VirtualView(m.context)
return origin + window.col_from_utf8(0, col)
return super().convert_column(line, col, m, vv)



reposition_match
----------------
.. code-block:: python

reposition_match(self, line, col, m, vv) -> tuple[int, int, int]

For linters that only report a column this method is called to find a suitable
region to highlight. Usually the :ref:`word<word_re>` at ``col`` is selected.
Returns ``(line, start, end)`` in code points.

``col`` has been converted and clamped, whereas ``m.col`` retains the raw adapter
column. This hook is not called when the linter already reports an ``end_col``.
Put selective unit conversion in :ref:`convert_column`, which covers both paths,
rather than converting an already-clamped column here.

9 changes: 9 additions & 0 deletions docs/linter_plugin.rst
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,15 @@ By default, SublimeLinter will run he linter in "stdin" mode, but you can change

Only ``message`` and ``line`` are mandatory fields here, but usually you also capture ``col``, ``error_type`` (e.g. "warning", "error"), and ``code`` (the name of the rule, e.g. "E302" or "no-console-log"). You can omit ``filename`` if the linter emits for only one file at a time anyways. You can also report ``end_line`` and ``end_col`` but that is seldom.

.. note::

If the tool reports UTF-8 byte or UTF-16 code-unit columns, declare
``column_unit = "utf8"`` or ``column_unit = "utf16"`` on the class.
Existing ``col`` / ``end_col`` groups and parsers can remain unchanged.
See :ref:`column_unit` and :ref:`convert_column` for mixed-unit tools
and source-window coordinates. Test through ``process_match`` with a
``VirtualView(source)``: parser-only tests cannot verify conversion.

.. note::

If you only report ``col``, SublimeLinter will select the word beginning at that column. What a word is, is defined by the :ref:`word_re` attribute.
Expand Down
78 changes: 67 additions & 11 deletions lint/linter.py
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,8 @@ def get_effective_lint_mode(settings) -> set[Reason]:


LEGACY_LINT_MATCH_DEF = ("match", "line", "col", "error", "warning", "message", "near")
ColumnUnit = Literal['codepoint', 'utf8', 'utf16']
COLUMN_UNITS: tuple[ColumnUnit, ...] = ('codepoint', 'utf8', 'utf16')
COMMON_CAPTURING_NAMES = (
"filename", "error_type", "code", "end_line", "end_col") + LEGACY_LINT_MATCH_DEF

Expand Down Expand Up @@ -213,6 +215,30 @@ def from_file(filename: str) -> VirtualView:
"""Return a VirtualView with the contents of file."""
return _virtual_view_from_file(filename, os.path.getmtime(filename))

def col_from_utf8(self, line: int, col: int) -> int:
"""Convert a zero-based byte offset to a code-point column.

Incomplete characters round down; offsets outside the line saturate.
"""
return self._col_from_units(line, col, 'utf8')

def col_from_utf16(self, line: int, col: int) -> int:
"""Convert a zero-based UTF-16 offset to a code-point column."""
return self._col_from_units(line, col, 'utf16')

def _col_from_units(self, line: int, col: int, unit: ColumnUnit) -> int:
text = self.select_line(line)
remaining = max(0, col)
for index, char in enumerate(text):
width = (
len(char.encode('utf-8', 'surrogatepass'))
if unit == 'utf8' else 2 if ord(char) > 0xffff else 1
)
if remaining < width:
return index
remaining -= width
return len(text)


@lru_cache(maxsize=128)
def _virtual_view_from_file(filename: str, mtime: float) -> VirtualView:
Expand Down Expand Up @@ -586,6 +612,12 @@ def __init__( # type: ignore[misc]
# END CLASS MUTATIONS

# BEGIN VALIDATION
if cls.column_unit not in COLUMN_UNITS:
logger.error(
f"{name} disabled, 'column_unit' must be one of {COLUMN_UNITS}."
)
cls.disabled = True

if not cls.cmd and cls.cmd is not None:
logger.error(
"{} disabled, 'cmd' must be specified."
Expand Down Expand Up @@ -708,6 +740,10 @@ class Linter(metaclass=LinterMeta):
# linter class should define this attribute accordingly.
line_col_base = (1, 1)

# Units used by the tool for col/end_col. This declares the output format,
# not an end-user setting. Conversion happens against source, before clamping.
column_unit: ColumnUnit = 'codepoint'

# If the linter executable cannot receive from stdin and requires a temp file,
# set this attribute to the suffix of the temp file (with or without leading '.').
# If the suffix needs to be mapped to the syntax of a file, you may make this
Expand Down Expand Up @@ -1380,24 +1416,28 @@ def process_match(self, m: LintMatch, vv: VirtualView) -> LintError | None:
.format(m.line + self.line_col_base[0])
)

# Convert local columns against the correct source, before clamping.
# Keep m unchanged: its columns remain the raw adapter coordinates.
end_line = line if m.end_line is None else max(line, min(m.end_line, vv.max_lines()))
col = None if m.col is None else self.convert_column(line, m.col, m, vv)
end_col = None if m.end_col is None else self.convert_column(end_line, m.end_col, m, vv)
line_region = vv.full_line_region(line)

if m.end_line is None and m.end_col is None:
_col = None if m.col is None else max(min(m.col, len(line_region) - 1), 0)
_col = None if col is None else max(min(col, len(line_region) - 1), 0)
line, col, end = self.reposition_match(line, _col, m, vv)
line_region = vv.full_line_region(line) # read again as `line` might have changed
region = sublime.Region(line_region.a + col, line_region.a + end)

else:
col = 0 if m.col is None else max(min(m.col, len(line_region) - 1), 0)
end_line = line if m.end_line is None else max(line, min(m.end_line, vv.max_lines()))
col = 0 if col is None else max(min(col, len(line_region) - 1), 0)
end_line_region = vv.line_region(end_line)
end_col = (
end = (
len(end_line_region)
if m.end_col is None
if end_col is None
else max(
col if end_line == line else 0,
min(m.end_col, len(end_line_region))
min(end_col, len(end_line_region))
)
)

Expand All @@ -1415,14 +1455,14 @@ def process_match(self, m: LintMatch, vv: VirtualView) -> LintError | None:
.format(m.end_line)
)

if m.end_col is not None:
if end_line == line and m.end_col < col:
if end_col is not None:
if end_line == line and end_col < col:
self.logger.warning(
"Reported end_col '{}' is before the start col '{}'."
.format(m.end_col, col)
.format(end_col, col)
)

region = sublime.Region(line_region.a + col, end_line_region.a + end_col)
region = sublime.Region(line_region.a + col, end_line_region.a + end)

# ensure a length of 1 but do not exceed eof (`size()`)
normalized_region = sublime.Region(
Expand All @@ -1441,6 +1481,21 @@ def process_match(self, m: LintMatch, vv: VirtualView) -> LintError | None:
"offending_text": offending_text
}

def convert_column(self, line: int, col: int, m: LintMatch, vv: VirtualView) -> int:
"""Convert one raw zero-based column to a code-point offset.

Called for each present col/end_col before clamping. `line` is its
bounded source row; `m` is the unchanged adapter output. Override for
mixed-unit tools or source-window coordinates, returning an integer.
"""
if self.column_unit == 'codepoint':
return col
if self.column_unit == 'utf8':
return vv.col_from_utf8(line, col)
if self.column_unit == 'utf16':
return vv.col_from_utf16(line, col)
raise ValueError(f'Unknown column unit: {self.column_unit!r}')

def get_error_type(self, error, warning):
if error:
return ERROR
Expand Down Expand Up @@ -1479,7 +1534,8 @@ def is_stdin_filename(filename: str) -> bool:
def reposition_match(self, line: int, col: Optional[int], m: LintMatch, vv: VirtualView) -> tuple[int, int, int]:
"""Chance to reposition the error.

Must return a tuple (line, start, end)
Must return a tuple (line, start, end), all in code points.
`col` has been converted and clamped; `m.col` retains the raw value.

The default implementation just finds a good `end` or range for the
given match. E.g. it uses `self.word_re` to select the whole word
Expand Down
Loading
Loading