sql.py: put_table(): log_exc(): Check if exception already caught before to avoid infinite loops

Added debug2redmine and helper file debug2redmine.csv

sql.py, db_xml.py: Removed unnecessary calls to sql_gen.clean_name() now that str() handles this automatically

sql_gen.py: sql_gen classes inherit from new base class BasicObject, whose str() calls clean_name() on the object's repr(). Changed the main debug-repr producing method to be repr() instead of str().

Moved clean_name() from sql.py to sql_gen.py because it's DB-general and so that it can be used by sql_gen.py without circular dependencies

db_xml.py: into_table_name(): Handle hierarchical tables specially by including their rank in the into table. Interpret any table with a value column as a function, regardless of out_table name.

sql.py: put_table(): Log "Default value column does not exist in mapping" error with level 2.1 so that it doesn't appear in Redmine output

db_xml.py: put_table(): Pass next as sql.put_table()'s default param now that it is supported

sql.py: put_table(): Changed default param to be an output column because that is what would be passed in by db_xml.put_table(), and because there is already a mapping that resolves that to a flattened input column

sql.py: put_table(): Added default param for the value or input column to use as the pkey for missing rows

sql.py: put_table(): Use single quotes rather than double quotes around strings where possible

db_xml.py: Added internal next param used by simplifyPath. put_table(): Refactored to use outer parent_ids_loc var and modify that as needed rather than having to pass parent_ids_loc as a param to put_table_().

sql.py: put_table(): When calling strings.as_*table(), pass custom ustr that removes col renames and adds double quotes on plain strings

strings.py: as_*table(): Added ustr param to override the method (by default ustr()) used to convert each value to a string

sql_gen.py: MockDb.esc_value(): Use new strings.repr_no_u()

strings.py: Added repr_no_u()

sql.py: clean_name(): Also remove '`' (which is used by MySQL)

sql.py: esc_name_by_module(): Use new sql_gen.esc_name()

sql_gen.py: Added esc_name() and use it in MockDb.esc_name()

sql.py: next_version(): Use special chars in version part of name string for clarity

sql.py: mk_insert_select(): embeddable: function_name is first line of query for clarity, and to reduce length from including the column names. This also fixes the problem of double quotes around column names in the previous function_name.

sql.py: esc_name_by_module(): Double embedded quotes to escape them instead of removing them

sql.py: put_table(): Use "-" to separate temp table suffixes from into table name

db_xml.py: into_table_name(): Format relational functions' into table names as a function call on the value column, using special chars for readability

sql.py: run_query(): Exception parsing: Use "(.+?)" wherever possible to match names containing special chars

sql.py: clean_name(): For clarity, just remove '"'s, so that "."s are preserved and show the path structure of the input name

db_xml.py: put_table(): sql.put_table(): Name the into table ...literal instead of ...value if the value column is a literal value

bin/map: Logging: log(): Remove extra debug info from DB query messages and format level 1.5 (summary) messages as Redmine list items

sql.py: put_table(): Renamed temp_prefix param to into and allow it to be a sql_gen.Table object. Use into directly as the pkeys table, and make its default value be `out_table.name+'_pkeys'`.

db_xml.py: put_table(): Pass custom temp_prefix to sql.put_table() for relational funcs, so that their value param's input column name is included in the temp table name

sql.py: put_table(): Added optional param temp_prefix for the prefix of generated temp tables

sql.py: put_table(): Made debug messages more self-documenting

sql.py: put_table(): Changed "Setting missing rows' pkeys to NULL" to "Setting pkeys of missing rows to NULL" to avoid having single quote in debug output, which messes up text editor SQL syntax highlighting

sql.py: Parsed exceptions: Use strings.as_tt() to format Python values

strings.py: Split as_table() into as_table() and as_inline_table() depending on whether the table needs to be inlined in an ordered list item or not

strings.py: Split as_table() into as_table() and as_inline_table() depending on whether the table needs to be inlined in an ordered list item or not

strings.py: as_table(): Changed to use

 formatting because Redmine tables can't be embedded in ordered lists without restarting the numbering

strings.py: as_table(): Fixed bug where table was not ended properly, by adding a space after the last \n and having rstrip() string only newlines

sql.py: mk_select(): Columns: Separate columns with newlines

sql.py: put_table(): Use new strings.as_table() to format mappings as tables

strings.py: Added as_tt() and as_table()

bin/map: Logging: log(): Strip trailing newlines from msg

strings.py: as_code(): Added multiline param to disable multiline formatted output

sql.py: put_table(): "Ignoring existing rows, comparing on" debug message: Wrap the mapping in strings.as_code() so it will have Redmine syntax-highlighting

sql.py: put_table(): "Putting columns" debug message: Wrap the mapping in strings.as_code() so it will have Redmine syntax-highlighting

sql.py: DbConn.run_query(): Query debug message: Wrap the query in strings.as_code() so it will have Redmine syntax-highlighting

strings.py: Added as_code()

sql.py: DbConn.run_query(): Prepend "DB query" before the query debug message so it can be identified as a DB query

db_xml.py: put_table(): Subset in_table: Document that in_table will be shadowed (hidden) by the created temp table, rather than versioned, now that the table is (almost) always created as a temp table

sql.py: Create temp items as permanent in autocommit mode rather than in debug mode so that temp items are only permanent if actually committing result. This ensures that the generated SQL in test mode matches what would actually get run in regular commit mode, and the SQL is only altered to make the temp items visible if actually debugging (autocommit mode).

sql.py, sql_gen.py: Reformatted generated SQL for presentability by adding newlines

sql.py: DbConn.run_query(): Put a newline before the query in the debug message so that multiline queries have all rows at the left edge rather than the first row prefixed by other text

sql.py: DbConn.run_query(): Don't put generated query debug message all on one line, so that embedded newlines are preserved

sql.py: Fixed bug where queries with versioned identifiers which threw an exception (not related to name collisions) were being output with a too-high log_level, because all exceptions were output with the higher exc_log_level, by making the following changes: DbConn.run_query(): Changed exc_log_level param to log_ignore_excs param so that only certain exceptions would cause the query to be output with a higher log_level. Moved the code that actual emits the query debug message from DbConn.run_query() to module-level run_query() so it would apply the log_ignore_excs filter after the exception had already been parsed into specific types.

Moved "Putting columns" debug message from db_xml.py put_table() to sql.py put_table() to put it in the same place as the other debug messages

sql_gen.py: Added remove_col_rename() and use it where `if isinstance(value, NamedCol): value = value.code` was used

sql_gen.py: CompareCond.to_str(): If left_value has been renamed as a NamedCol, unwrap it

sql_gen.py: Join.to_str(): Fixed bug where USING should be used if all columns are join_same_not_null, rather than join_same, because USING uses plain = for comparison. sql.py: put_table(): input_joins now can use sql_gen.join_same_not_null in order to use USING syntax.

db_xml.py: put_table(): Output debug messages with a level of 1.5 to match sql.put_table()'s level for summary messages

bin/map: Fixed bug where verbosity needed to be 1 outside of test mode so that profiling and errors stats would be printed at end of import. Verbosity defaults to 0.5 rather than 1 in test mode so profiling and errors stats do not clutter up the test output when running automated tests.

bin/map: Only display verbose_errors in test mode, but with any nonzero verbosity. They should not be displayed outside of test mode because verbose errors make the log files huge.

bin/map: Renamed verbose param to verbosity because it's now a number, not a boolean

bin/map: Removed no longer used debug param (verbose=2 is used instead)

bin/map: Fixed bug where verbose_errors' default value depended on debug var, which was not yet set. Removed verbose_errors param and instead turn verbose_errors on whenever verbosity >= 1. Verbosity defaults to 1 in test mode.

bin/map: Logging: Don't set sql.run_raw_query.debug, because it is not used anymore (sql.connect(log_debug=...) is used instead)

bin/map: Logging: Print debug messages (level > 1) prefixed with their level, to distinguish higher- and lower-level debug messages

sql.py: put_table(): Only display warning for exceptions with no handler (which are unexpected), not missing mappings for NOT NULL columns (which are normal in datasources without those columns)

sql.py: put_table(): Log summarizing debug messages with a level of 1.5 so they will be displayed even when the major SQL queries (which have a level of 2) are not shown

bin/map: Provide a log_debug() function to sql.connect() if verbosity > 1 rather than >= 2, to support fractional verbosities

sql.py: log_debug_none: Fixed bug where needed to take kw arg level to work with verbosity-based logging

bin/map: Allow fractional verbosity values

sql.py: Functions that version created tables, functions, etc. if they already exist: Use (default) exc_log_level=4 to hide the unsuccessful attempts to create items that already exist and show only the successful attempt

sql.py: DbConn.run_query(): Added exc_log_level param to specify a different log_level if the query throws an exception. This will useful for functions that version created tables, functions, etc. if they already exist.

sql.py: DbConn.run_query(): Removed no longer accurate doc comment, because that functionality is now in module-level run_query()

sql.py: Specify log_levels for minor queries so they can be excluded from the debug output

sql.py: select(): Pass log_level to run_query()

sql.py: DbConn.run_query(): Added log_level param and pass it to self.log_debug(). run_query(): Pass extra kw_args to DbConn.run_query() (via run_raw_query()) so that caller can specify log_level.

sql.py: run_query_into(): Fixed bug where "temporary tables cannot specify a schema name"

bin/map: Switched to verbosity-level-based system of logging. verbose is now an integer, and debug sets the minimum verbosity to 2.

input.Makefile: Configuration: Removed debug var since it's not used in the Makefile

db_xml.py: put_table(): put_table_(): Fixed bug where row_ins_ct_ref needed to be passed recursively to put_table() as keyword arg, because the in_row_ct_ref is not passed recursively

db_xml.py: put_table(): _simplifyPath: Parse "next" XPath param to extract col name of next level's pkey

bin/map: by_col: xml_func.strip(): Don't remove _simplifyPath because it is now handled by db_xml.put_table()

db_xml.py: put_table(): Added basic special handling for structural XML functions, which for now just skips the function

xml_func.py: strip(): Added preserve param for XML functions not to remove

db_xml.py: put_table(): Handle forward pointers in translation-to-sql_gen step instead of in XML-tree-parsing step, so that special handling for structural XML functions can use the parsed tree before any sql.put_table() processing takes place

xml_dom.py: Added is_node()

sql.py: table_row_count(): Pass start=0 to mk_select() to avoid "SELECT statement missing a WHERE, LIMIT, or OFFSET clause" warnings

sql.py: put_table(): Handle unknown exceptions by returning NULL for all rows. Refactored Missing mapping for NOT NULL column handling to use new helper function remove_all_rows().

sql.py: put_table(): Assert that insert_out_pkeys and insert_in_pkeys have same row count. Assert that pkeys and in_table have same row count.

db_xml.py: put_table(): Use new sql.table_row_count()

sql.py: Added table_row_count()

db_xml.py: put_table(): Use new sql_gen.row_count

sql_gen.py: Added row_count

db_xml.py: put_table(): Count # rows and update in_row_ct_ref once all columns have been processed. Don't pass in_row_ct_ref to recursive calls because it should only be increased once.

db_xml.py: put_table(): Added in_row_ct_ref param to store the # of input rows processed. Renamed row_ct_ref param to row_ins_ct_ref to distinguish it from new in_row_ct_ref param.

sql_gen.py: MockDb.esc_name(): Don't use sql.esc_name_by_module() to avoid circular dependency on sql module

sql.py: put_table(): Factored out mk_select() calls in calls to run_query_into_pkeys() into new helper function insert_into_pkeys()

sql.py: put_table(): run_query_into_pkeys() calls use order_by=None in their select statements because there is a pkey, so order (row #) does not matter

db_xml.py: put_table(): Subset in_table if limit != None or start != 0. start param defaults to 0 again to avoid subsetting the table when starting from row 0 (with no limit).