Moved clean_name() from sql.py to sql_gen.py because it's DB-general and so that it can be used by sql_gen.py without circular dependencies
db_xml.py: into_table_name(): Handle hierarchical tables specially by including their rank in the into table. Interpret any table with a value column as a function, regardless of out_table name.
sql.py: put_table(): Log "Default value column does not exist in mapping" error with level 2.1 so that it doesn't appear in Redmine output
db_xml.py: put_table(): Pass next as sql.put_table()'s default param now that it is supported
sql.py: put_table(): Changed default param to be an output column because that is what would be passed in by db_xml.put_table(), and because there is already a mapping that resolves that to a flattened input column
sql.py: put_table(): Added default param for the value or input column to use as the pkey for missing rows
sql.py: put_table(): Use single quotes rather than double quotes around strings where possible
db_xml.py: Added internal next param used by simplifyPath. put_table(): Refactored to use outer parent_ids_loc var and modify that as needed rather than having to pass parent_ids_loc as a param to put_table_().
sql.py: put_table(): When calling strings.as_*table(), pass custom ustr that removes col renames and adds double quotes on plain strings
strings.py: as_*table(): Added ustr param to override the method (by default ustr()) used to convert each value to a string
sql_gen.py: MockDb.esc_value(): Use new strings.repr_no_u()
strings.py: Added repr_no_u()
sql.py: clean_name(): Also remove '`' (which is used by MySQL)
sql.py: esc_name_by_module(): Use new sql_gen.esc_name()
sql_gen.py: Added esc_name() and use it in MockDb.esc_name()
sql.py: next_version(): Use special chars in version part of name string for clarity
sql.py: mk_insert_select(): embeddable: function_name is first line of query for clarity, and to reduce length from including the column names. This also fixes the problem of double quotes around column names in the previous function_name.
sql.py: esc_name_by_module(): Double embedded quotes to escape them instead of removing them
sql.py: put_table(): Use "-" to separate temp table suffixes from into table name
db_xml.py: into_table_name(): Format relational functions' into table names as a function call on the value column, using special chars for readability
sql.py: run_query(): Exception parsing: Use "(.+?)" wherever possible to match names containing special chars
sql.py: clean_name(): For clarity, just remove '"'s, so that "."s are preserved and show the path structure of the input name
db_xml.py: put_table(): sql.put_table(): Name the into table ...literal instead of ...value if the value column is a literal value
bin/map: Logging: log(): Remove extra debug info from DB query messages and format level 1.5 (summary) messages as Redmine list items
sql.py: put_table(): Renamed temp_prefix param to into and allow it to be a sql_gen.Table object. Use into directly as the pkeys table, and make its default value be `out_table.name+'_pkeys'`.
db_xml.py: put_table(): Pass custom temp_prefix to sql.put_table() for relational funcs, so that their value param's input column name is included in the temp table name
sql.py: put_table(): Added optional param temp_prefix for the prefix of generated temp tables
sql.py: put_table(): Made debug messages more self-documenting
sql.py: put_table(): Changed "Setting missing rows' pkeys to NULL" to "Setting pkeys of missing rows to NULL" to avoid having single quote in debug output, which messes up text editor SQL syntax highlighting
sql.py: Parsed exceptions: Use strings.as_tt() to format Python values
strings.py: Split as_table() into as_table() and as_inline_table() depending on whether the table needs to be inlined in an ordered list item or not
strings.py: as_table(): Changed to use
formatting because Redmine tables can't be embedded in ordered lists without restarting the numbering
strings.py: as_table(): Fixed bug where table was not ended properly, by adding a space after the last \n and having rstrip() string only newlines
sql.py: mk_select(): Columns: Separate columns with newlines
sql.py: put_table(): Use new strings.as_table() to format mappings as tables
strings.py: Added as_tt() and as_table()
bin/map: Logging: log(): Strip trailing newlines from msg
strings.py: as_code(): Added multiline param to disable multiline formatted output
sql.py: put_table(): "Ignoring existing rows, comparing on" debug message: Wrap the mapping in strings.as_code() so it will have Redmine syntax-highlighting
sql.py: put_table(): "Putting columns" debug message: Wrap the mapping in strings.as_code() so it will have Redmine syntax-highlighting
sql.py: DbConn.run_query(): Query debug message: Wrap the query in strings.as_code() so it will have Redmine syntax-highlighting
strings.py: Added as_code()
sql.py: DbConn.run_query(): Prepend "DB query" before the query debug message so it can be identified as a DB query
db_xml.py: put_table(): Subset in_table: Document that in_table will be shadowed (hidden) by the created temp table, rather than versioned, now that the table is (almost) always created as a temp table
sql.py: Create temp items as permanent in autocommit mode rather than in debug mode so that temp items are only permanent if actually committing result. This ensures that the generated SQL in test mode matches what would actually get run in regular commit mode, and the SQL is only altered to make the temp items visible if actually debugging (autocommit mode).
sql.py, sql_gen.py: Reformatted generated SQL for presentability by adding newlines
sql.py: DbConn.run_query(): Put a newline before the query in the debug message so that multiline queries have all rows at the left edge rather than the first row prefixed by other text
sql.py: DbConn.run_query(): Don't put generated query debug message all on one line, so that embedded newlines are preserved
sql.py: Fixed bug where queries with versioned identifiers which threw an exception (not related to name collisions) were being output with a too-high log_level, because all exceptions were output with the higher exc_log_level, by making the following changes: DbConn.run_query(): Changed exc_log_level param to log_ignore_excs param so that only certain exceptions would cause the query to be output with a higher log_level. Moved the code that actual emits the query debug message from DbConn.run_query() to module-level run_query() so it would apply the log_ignore_excs filter after the exception had already been parsed into specific types.
Moved "Putting columns" debug message from db_xml.py put_table() to sql.py put_table() to put it in the same place as the other debug messages
sql_gen.py: Added remove_col_rename() and use it where `if isinstance(value, NamedCol): value = value.code` was used
sql_gen.py: CompareCond.to_str(): If left_value has been renamed as a NamedCol, unwrap it
sql_gen.py: Join.to_str(): Fixed bug where USING should be used if all columns are join_same_not_null, rather than join_same, because USING uses plain = for comparison. sql.py: put_table(): input_joins now can use sql_gen.join_same_not_null in order to use USING syntax.
db_xml.py: put_table(): Output debug messages with a level of 1.5 to match sql.put_table()'s level for summary messages
bin/map: Fixed bug where verbosity needed to be 1 outside of test mode so that profiling and errors stats would be printed at end of import. Verbosity defaults to 0.5 rather than 1 in test mode so profiling and errors stats do not clutter up the test output when running automated tests.
bin/map: Only display verbose_errors in test mode, but with any nonzero verbosity. They should not be displayed outside of test mode because verbose errors make the log files huge.
bin/map: Renamed verbose param to verbosity because it's now a number, not a boolean
bin/map: Removed no longer used debug param (verbose=2 is used instead)
bin/map: Fixed bug where verbose_errors' default value depended on debug var, which was not yet set. Removed verbose_errors param and instead turn verbose_errors on whenever verbosity >= 1. Verbosity defaults to 1 in test mode.
bin/map: Logging: Don't set sql.run_raw_query.debug, because it is not used anymore (sql.connect(log_debug=...) is used instead)
bin/map: Logging: Print debug messages (level > 1) prefixed with their level, to distinguish higher- and lower-level debug messages
sql.py: put_table(): Only display warning for exceptions with no handler (which are unexpected), not missing mappings for NOT NULL columns (which are normal in datasources without those columns)
sql.py: put_table(): Log summarizing debug messages with a level of 1.5 so they will be displayed even when the major SQL queries (which have a level of 2) are not shown
bin/map: Provide a log_debug() function to sql.connect() if verbosity > 1 rather than >= 2, to support fractional verbosities
sql.py: log_debug_none: Fixed bug where needed to take kw arg level to work with verbosity-based logging
bin/map: Allow fractional verbosity values
sql.py: Functions that version created tables, functions, etc. if they already exist: Use (default) exc_log_level=4 to hide the unsuccessful attempts to create items that already exist and show only the successful attempt
sql.py: DbConn.run_query(): Added exc_log_level param to specify a different log_level if the query throws an exception. This will useful for functions that version created tables, functions, etc. if they already exist.
sql.py: DbConn.run_query(): Removed no longer accurate doc comment, because that functionality is now in module-level run_query()
sql.py: Specify log_levels for minor queries so they can be excluded from the debug output
sql.py: select(): Pass log_level to run_query()
sql.py: DbConn.run_query(): Added log_level param and pass it to self.log_debug(). run_query(): Pass extra kw_args to DbConn.run_query() (via run_raw_query()) so that caller can specify log_level.
sql.py: run_query_into(): Fixed bug where "temporary tables cannot specify a schema name"
bin/map: Switched to verbosity-level-based system of logging. verbose is now an integer, and debug sets the minimum verbosity to 2.
input.Makefile: Configuration: Removed debug var since it's not used in the Makefile
db_xml.py: put_table(): put_table_(): Fixed bug where row_ins_ct_ref needed to be passed recursively to put_table() as keyword arg, because the in_row_ct_ref is not passed recursively
db_xml.py: put_table(): _simplifyPath: Parse "next" XPath param to extract col name of next level's pkey
bin/map: by_col: xml_func.strip(): Don't remove _simplifyPath because it is now handled by db_xml.put_table()
db_xml.py: put_table(): Added basic special handling for structural XML functions, which for now just skips the function
xml_func.py: strip(): Added preserve param for XML functions not to remove
db_xml.py: put_table(): Handle forward pointers in translation-to-sql_gen step instead of in XML-tree-parsing step, so that special handling for structural XML functions can use the parsed tree before any sql.put_table() processing takes place
xml_dom.py: Added is_node()
sql.py: table_row_count(): Pass start=0 to mk_select() to avoid "SELECT statement missing a WHERE, LIMIT, or OFFSET clause" warnings
sql.py: put_table(): Handle unknown exceptions by returning NULL for all rows. Refactored Missing mapping for NOT NULL column handling to use new helper function remove_all_rows().
sql.py: put_table(): Assert that insert_out_pkeys and insert_in_pkeys have same row count. Assert that pkeys and in_table have same row count.
db_xml.py: put_table(): Use new sql.table_row_count()
sql.py: Added table_row_count()
db_xml.py: put_table(): Use new sql_gen.row_count
sql_gen.py: Added row_count
db_xml.py: put_table(): Count # rows and update in_row_ct_ref once all columns have been processed. Don't pass in_row_ct_ref to recursive calls because it should only be increased once.
db_xml.py: put_table(): Added in_row_ct_ref param to store the # of input rows processed. Renamed row_ct_ref param to row_ins_ct_ref to distinguish it from new in_row_ct_ref param.
sql_gen.py: MockDb.esc_name(): Don't use sql.esc_name_by_module() to avoid circular dependency on sql module
sql.py: put_table(): Factored out mk_select() calls in calls to run_query_into_pkeys() into new helper function insert_into_pkeys()
sql.py: put_table(): run_query_into_pkeys() calls use order_by=None in their select statements because there is a pkey, so order (row #) does not matter
db_xml.py: put_table(): Subset in_table if limit != None or start != 0. start param defaults to 0 again to avoid subsetting the table when starting from row 0 (with no limit).
db_xml.py: put_table(): Don't pass limit, start recursively, because the table subsetting will happen only once in the first invocation of the function. Moved limit, start params to end since they are not passed recursively. start param no longer defaults to 0 because this is not needed since sql.put_table() now sets start to 0 where needed.
sql.py: put_table(): Removed limit and start params because they were never fully implemented, and because it's simpler to just have the caller subset their input table
lists.py: Added uniqify()
sql.py: Moved mk_flatten_mapping(), flatten() to Basic queries section since they don't involve database structure info