MDEV-41962 Fallback SQL parser for vendor-dialect statements offloaded to an external analytical engine - #6141

A new plugin type, MariaDB_SQL_EXTENSION_PLUGIN, for plugins that handle
SQL statements the server's parser rejects. This commit adds only the
type and its descriptor (include/mysql/plugin_sql_extension.h): a list
of keyword prefixes or a recognize() callback (one or the other),
execute(), and a priority among recognizers. The registry that consults
such plugins after a parse failure follows in a later commit; until then
a plugin of this type can be declared and installed but is never called.

sql_plugin.cc: the type's name and interface versions, and a new
plugin_lock_by_name_ready() next to the existing plugin_lock_ready():
both return NULL unless the plugin is READY, so a plugin that UNINSTALL
has marked deleted while still in use is not handed out again.

Part of REST-5 (SQL extension registry for server plugins).

sql/sql_extension_matcher.{h,cc}: a trie whose edges are uppercased
keywords rather than bytes, used by the SQL extension registry to match
the prefixes plugins declare ("CREATE REST", "CREATE OR REPLACE REST").
The word reader skips whitespace and the three comment forms with the
connection charset and consumes multi-byte characters whole, never
splitting one. Matching is one sweep over the first words of a
statement, each word a binary search among the keywords that may follow
the previous one, so the cost grows with the length of the matched
prefix and only logarithmically with the number of registered prefixes.
The trie holds claims (a prefix owned by one plugin, with overlap
detection between owners) and removes one owner's entries without
rebuilding the others'.

sql_ext_statement_length() finds where the statement at the start of a
text ends: its first ';' outside quoted strings, quoted identifiers and
comments, read as the server lexer reads them (doubled quotes, backslash
escapes unless NO_BACKSLASH_ESCAPES, "..." an identifier under
ANSI_QUOTES, multi-byte characters whole; executable comments count as
comments, as for the prefix walk). The registry ends a prefix plugin's
statement there, so no plugin needs a scanner of its own.

Keywords, prefixes and node arrays are allocated with my_malloc() and
Dynamic_array under a PSI memory key the registry provides, so that
memory is instrumented; the nodes come from new (std::nothrow).
Allocation failures are reported, not thrown.

Unit tests in unittest/sql/sql_extension_matcher-t.cc.

Part of REST-5 (SQL extension registry for server plugins).

include/mysql/service_sql_extension.h, service sql_extension_service
0x0100, included from plugin_sql_extension.h (not from services.h, so
only extension plugins see it): what a plugin's execute() uses to
answer a statement.

- sql_extension_send_result_set(): a materialized result set of STRING,
  BINARY, INTEGER, UNSIGNED, DOUBLE, DECIMAL, DATE, TIME, DATETIME or
  JSON columns, every cell given as text and checked before anything is
  sent (ER_SQL_EXTENSION_INVALID_RESULT names the cell or the column).
  The metadata follows the cells: string widths in characters, DECIMAL
  precision and scale, fractional seconds, nullability; JSON carries
  format=json. Rows go through the typed Protocol::store_*() functions,
  so the binary protocol gets native values. Each row counts as a row
  sent (Rows_sent of the slow log, ROWS_SENT of the performance schema).
- sql_extension_send_ok(): an OK with affected rows and a message.
- sql_extension_check_access(): the client's privileges on a database
  or a table, with bits of its own that map exactly to DB_ACLS (a
  static_assert keeps them so); the database-level check computes the
  effective privileges by name, as fill_effective_table_privileges()
  does, since check_access() would read the name as a GRANT pattern.
- sql_extension_client_flags(): whether the client accepts several
  results, and whether its packet may hold several statements.
- Several results per statement, as CALL sends them: the final packet
  of the latest result is held back and sent with
  SERVER_MORE_RESULTS_EXISTS when the next one starts; one result is
  sent as before. An error after results follows them; a client that
  does not accept several results gets the error in place of the end of
  the result it was sent, as when a SELECT fails after some rows.
- Notes and warnings are added with my_printf_error() and ME_NOTE or
  ME_WARNING; the service has no function for them.
- THD::send_kill_message() returns whether it raised an error (LIMIT
  ROWS EXAMINED stops a statement without one), so that a service
  function called after a kill fails with the kill error; its message
  comes from the new private THD::killed_message().

Test plugin: ROWS, EMPTY, NULLS, TYPES, BAD <type> <text>, BAD2, BADSET,
BADTHD, INFO, WARN, ACCESS, TWICE, MULTI, FLAGS, a second recognizer
with priority 10, a prefix plugin with a priority, which INSTALL
refuses, and a plugin whose init() fails. Tests: install (both refused,
a failed init() not followed by deinit()), install_startup (all the
test plugins at startup), plain_service (results through the plain
SQL service), service (results, metadata, also with no rows
or only NULLs, prepared statements, every column type at its limits, a
DECIMAL column too wide only across rows, malformed result sets, calls
on the wrong THD, access and name checks, every invalid argument),
service_lctn (access checks with lower_case_table_names=1), results
(several results), results_debug (clients without multi-results, a
result the plugin set outside the service), priority (recognizers by
priority, not by installation order; a claimed prefix before any
recognizer), perfschema (rows of result sets counted as rows sent).

Part of REST-5 (SQL extension registry for server plugins).

Most extension plugins implement their statements by running ordinary
SQL. The framework makes that correct for every plugin through a new
mode of the plugin SQL service's local connection, rather than leaving
privileges, logging and KILL to each plugin.

- Protocol_local gets the serial of the extension statement it serves
  (as_client() when non-zero; sql_prepare.cc, entry point
  mysql_real_connect_local_as_client()): the SQL runs on the client's
  own THD with the client's security context, so its grants apply; with
  the client's sql_log_bin, so changes replicate as plain SQL and a
  replica needs no plugin; in the client's transaction; under LOCK
  TABLES with the client's locked tables, like SQL in a stored
  procedure; a pending KILL QUERY fails the SQL instead of being reset;
  an error the handler raised before running the SQL is not cleared;
  and each statement counts its own LIMIT ROWS EXAMINED. Outside an
  extension statement the entry point refuses, rather than fall back to
  the plain service. The existing behaviour of the service is untouched
  for its current users.
- sql_extension_sql_as_client(thd): true while the THD executes SQL on
  behalf of an extension statement, derived from the state the nesting
  guard keeps (no new THD field). Such SQL leaves the statement timer to
  the outer statement, as a stored procedure's statements leave it, so
  max_statement_time covers the whole extension statement. It leaves the
  kill state set too, which a stored procedure's statements do not do:
  the extension statement, and any SQL its handler runs after the kill,
  fail with the kill error, even when the handler ignores the error its
  SQL got; THD::killed_error() gives that error for the handle. This
  holds for the as-client connection, for forwarded SQL and for the
  plain SQL service when a handler uses it on the client's THD; the
  private THD that service creates under LOCK TABLES is not the
  client's and is not affected.
- Likewise, SQL whose error rolls back the client's transaction (a
  deadlock, a lock wait timeout with innodb_rollback_on_timeout) fails
  the extension statement with that error, even when the handler drops
  it, so the client knows its transaction is gone:
  trans_rollback_implicit() notes the error through
  sql_extension_note_implicit_rollback() when an engine asked for the
  rollback (not for CHECK TABLE of a corrupt table, which succeeds). A
  handler's own error is kept, as for a kill, and a stored program
  whose handler takes the error and does not let it out succeeds, as
  when the client calls it (sql_extension_note_error_handled(),
  sql_extension_note_sql_done()).
- sql_extension_connect_as_client(): that connection for a plugin. A
  connection kept past its statement is refused, by a serial unique to
  each extension statement of the server, and so is one used from
  another thread.
- sql_extension_forward_sql(): runs one statement as the client on the
  client's protocol (execute_sql_as_client()), so the client receives
  the server's own result sets, metadata and OK, including every result
  of a CALL; its warnings and error become the extension statement's.
  select_send::send_eof() leaves the final packet to the extension
  statement.
- Two guards of the earlier commits are now tested with SQL a handler
  runs. An extension statement claimed by that SQL is refused with
  ER_SQL_EXTENSION_NESTED, also through EXECUTE of a prepared one and
  for the plain SQL service's private THD under LOCK TABLES: the guard
  is a thread-local on purpose. And nothing a handler runs is stored by
  the query cache, which matters now that forwarded SELECTs reach the
  client: a carrier whose text starts like a SELECT was looked up before
  the parse, and the first forwarded SELECT would otherwise be stored
  under the carrier's text and served to the next client sending it
  without the plugin.

Test plugin: SQL, SQL2, LOCAL SQL, LOCAL SQL2, KEEP, REUSE, THREAD,
[LOCAL] SWALLOW [OK|ROWS] (also of a script), [LOCAL] RAISE, NOTHD,
BADCALL, FORWARD, SCRIPT; WITH SQLEXT also serves as a carrier the
query cache looks up.

Tests:
- service: handler SQL with the client's grants, current database,
  transaction and LOCK TABLES; SQL that changes the session or the
  transaction (COMMIT, USE, LOCK TABLES...); EXECUTE of an extension
  statement refused; KILL QUERY, KILL CONNECTION and max_statement_time,
  also for a handler that ignores the kill error or goes on after it; a
  deadlock that rolls back the client's transaction, dropped by the
  handler (in a transaction and in autocommit, after an OK or rows,
  through the plain service), replaced by the handler's own error, or
  handled by a stored procedure (also by an outer procedure's handler,
  through the forwarding service too, and not undone by a later
  unrelated error), but not when the procedure's handler re-raises it or
  another error, nor by a later error of the same number; a lock wait
  timeout without innodb_rollback_on_timeout, which leaves the
  transaction; CHECK TABLE of a corrupt table; an error raised before
  the handler's SQL; LIMIT ROWS EXAMINED; a connection kept past its
  statement or used from another thread; invalid calls, also from a
  thread with no THD.
- service_threadpool: a kept connection under the thread pool.
- service_debug: SQL a handler runs after the timeout does not run.
- service_kill: KILL QUERY through the plain SQL service too.
- results: forwarded results, also through the binary protocol, their
  insert id, results of extension statements inside a CALL, scripts,
  KILL and max_statement_time of forwarded SQL.
- results_debug: a forwarded CALL or a script for a client without
  multi-results, KILL between two statements of a script.
- service_binlog, rpl: handler SQL is logged as plain SQL and applied
  by a replica without the plugin, also when the handler fails after
  changing data.
- galera.galera_sql_extension: handler SQL replicates as plain SQL,
  also when the handler fails after changing data; INSTALL PLUGIN
  replicates as DDL, so every node has the plugin.
- rollback_on_timeout: a lock wait timeout with
  innodb_rollback_on_timeout fails the statement, even when the
  handler drops it, in a transaction and in autocommit, unless a stored
  procedure handles it.
- galera.galera_sql_extension_bf_abort: a transaction aborted by
  another node's while handler SQL runs gets ER_LOCK_DEADLOCK, as the
  same SQL sent directly, also when the handler drops the error.
- qcache; nesting in dispatch; UNINSTALL during a statement in
  lifecycle.

Part of REST-5 (SQL extension registry for server plugins).