sql.py, sql_gen.py: Reformatted generated SQL for presentability by adding newlines

sql.py: DbConn.run_query(): Put a newline before the query in the debug message so that multiline queries have all rows at the left edge rather than the first row prefixed by other text

sql.py: DbConn.run_query(): Don't put generated query debug message all on one line, so that embedded newlines are preserved

sql.py: Fixed bug where queries with versioned identifiers which threw an exception (not related to name collisions) were being output with a too-high log_level, because all exceptions were output with the higher exc_log_level, by making the following changes: DbConn.run_query(): Changed exc_log_level param to log_ignore_excs param so that only certain exceptions would cause the query to be output with a higher log_level. Moved the code that actual emits the query debug message from DbConn.run_query() to module-level run_query() so it would apply the log_ignore_excs filter after the exception had already been parsed into specific types.

Moved "Putting columns" debug message from db_xml.py put_table() to sql.py put_table() to put it in the same place as the other debug messages

sql_gen.py: Added remove_col_rename() and use it where `if isinstance(value, NamedCol): value = value.code` was used

sql_gen.py: CompareCond.to_str(): If left_value has been renamed as a NamedCol, unwrap it

sql_gen.py: Join.to_str(): Fixed bug where USING should be used if all columns are join_same_not_null, rather than join_same, because USING uses plain = for comparison. sql.py: put_table(): input_joins now can use sql_gen.join_same_not_null in order to use USING syntax.

db_xml.py: put_table(): Output debug messages with a level of 1.5 to match sql.put_table()'s level for summary messages

bin/map: Fixed bug where verbosity needed to be 1 outside of test mode so that profiling and errors stats would be printed at end of import. Verbosity defaults to 0.5 rather than 1 in test mode so profiling and errors stats do not clutter up the test output when running automated tests.

bin/map: Only display verbose_errors in test mode, but with any nonzero verbosity. They should not be displayed outside of test mode because verbose errors make the log files huge.

bin/map: Renamed verbose param to verbosity because it's now a number, not a boolean

bin/map: Removed no longer used debug param (verbose=2 is used instead)

bin/map: Fixed bug where verbose_errors' default value depended on debug var, which was not yet set. Removed verbose_errors param and instead turn verbose_errors on whenever verbosity >= 1. Verbosity defaults to 1 in test mode.

bin/map: Logging: Don't set sql.run_raw_query.debug, because it is not used anymore (sql.connect(log_debug=...) is used instead)

bin/map: Logging: Print debug messages (level > 1) prefixed with their level, to distinguish higher- and lower-level debug messages

sql.py: put_table(): Only display warning for exceptions with no handler (which are unexpected), not missing mappings for NOT NULL columns (which are normal in datasources without those columns)

sql.py: put_table(): Log summarizing debug messages with a level of 1.5 so they will be displayed even when the major SQL queries (which have a level of 2) are not shown

bin/map: Provide a log_debug() function to sql.connect() if verbosity > 1 rather than >= 2, to support fractional verbosities

sql.py: log_debug_none: Fixed bug where needed to take kw arg level to work with verbosity-based logging

bin/map: Allow fractional verbosity values

sql.py: Functions that version created tables, functions, etc. if they already exist: Use (default) exc_log_level=4 to hide the unsuccessful attempts to create items that already exist and show only the successful attempt

sql.py: DbConn.run_query(): Added exc_log_level param to specify a different log_level if the query throws an exception. This will useful for functions that version created tables, functions, etc. if they already exist.

sql.py: DbConn.run_query(): Removed no longer accurate doc comment, because that functionality is now in module-level run_query()

sql.py: Specify log_levels for minor queries so they can be excluded from the debug output

sql.py: select(): Pass log_level to run_query()

sql.py: DbConn.run_query(): Added log_level param and pass it to self.log_debug(). run_query(): Pass extra kw_args to DbConn.run_query() (via run_raw_query()) so that caller can specify log_level.

sql.py: run_query_into(): Fixed bug where "temporary tables cannot specify a schema name"

bin/map: Switched to verbosity-level-based system of logging. verbose is now an integer, and debug sets the minimum verbosity to 2.

input.Makefile: Configuration: Removed debug var since it's not used in the Makefile

db_xml.py: put_table(): put_table_(): Fixed bug where row_ins_ct_ref needed to be passed recursively to put_table() as keyword arg, because the in_row_ct_ref is not passed recursively

db_xml.py: put_table(): _simplifyPath: Parse "next" XPath param to extract col name of next level's pkey

bin/map: by_col: xml_func.strip(): Don't remove _simplifyPath because it is now handled by db_xml.put_table()

db_xml.py: put_table(): Added basic special handling for structural XML functions, which for now just skips the function

xml_func.py: strip(): Added preserve param for XML functions not to remove

db_xml.py: put_table(): Handle forward pointers in translation-to-sql_gen step instead of in XML-tree-parsing step, so that special handling for structural XML functions can use the parsed tree before any sql.put_table() processing takes place

xml_dom.py: Added is_node()

sql.py: table_row_count(): Pass start=0 to mk_select() to avoid "SELECT statement missing a WHERE, LIMIT, or OFFSET clause" warnings

sql.py: put_table(): Handle unknown exceptions by returning NULL for all rows. Refactored Missing mapping for NOT NULL column handling to use new helper function remove_all_rows().

sql.py: put_table(): Assert that insert_out_pkeys and insert_in_pkeys have same row count. Assert that pkeys and in_table have same row count.

db_xml.py: put_table(): Use new sql.table_row_count()

sql.py: Added table_row_count()

db_xml.py: put_table(): Use new sql_gen.row_count

sql_gen.py: Added row_count

db_xml.py: put_table(): Count # rows and update in_row_ct_ref once all columns have been processed. Don't pass in_row_ct_ref to recursive calls because it should only be increased once.

db_xml.py: put_table(): Added in_row_ct_ref param to store the # of input rows processed. Renamed row_ct_ref param to row_ins_ct_ref to distinguish it from new in_row_ct_ref param.

sql_gen.py: MockDb.esc_name(): Don't use sql.esc_name_by_module() to avoid circular dependency on sql module

sql.py: put_table(): Factored out mk_select() calls in calls to run_query_into_pkeys() into new helper function insert_into_pkeys()

sql.py: put_table(): run_query_into_pkeys() calls use order_by=None in their select statements because there is a pkey, so order (row #) does not matter

db_xml.py: put_table(): Subset in_table if limit != None or start != 0. start param defaults to 0 again to avoid subsetting the table when starting from row 0 (with no limit).

db_xml.py: put_table(): Don't pass limit, start recursively, because the table subsetting will happen only once in the first invocation of the function. Moved limit, start params to end since they are not passed recursively. start param no longer defaults to 0 because this is not needed since sql.put_table() now sets start to 0 where needed.

sql.py: put_table(): Removed limit and start params because they were never fully implemented, and because it's simpler to just have the caller subset their input table

lists.py: Added uniqify()

sql.py: Moved mk_flatten_mapping(), flatten() to Basic queries section since they don't involve database structure info

sql.py: put_table(): Use single quotes rather than double quotes around strings where possible

schemas/functions.sql, vegbien.sql: Changed CAST-related relational functions to return NULL on data exceptions and convert the exceptions to warnings. This helps column-based import by mapping invalid values to NULL instead of aborting the whole query on the first invalid value.

sql.py: index_col(): Cache the query so it doesn't try to add an index on the same column multiple times

sql.py mk_select(), sql_gen.py Join.to_str(): Fixed bug where conditions needed to be wrapped in () before being AND-ed together to ensure the proper operator precedence

sql.py: put_table(): Add index on columns with invalid values to enable fast filtering

sql.py: Added index_col()

sql.py: put_table(): Add pkey on returned pkeys table to enable fast joins

sql.py: Added index_pkey()

sql.py: mk_update(): When running sql_gen.to_name_only_col(), check that the col's table is table

sql.py: put_table(): Renamed pkeys to insert_pkeys to distinguish them from the full set of pkeys on the input table

sql.py: put_table(): FunctionValueException: Change invalid values to NULL using UPDATE instead of filtering them out using WHERE, to avoid adding lots of conditions to the SELECT statement

sql.py: Added mk_update() and update()

sql_gen.py: Added to_name_only_col()

sql_gen.py: Added as_Value()

sql.py: mk_select(): conds: Use new sql_gen.ColValueCond instead of sql_gen.as_ValueCond(). Documented that Code and ValueCond are sql_gen objects.

sql_gen.py: Added ColValueCond

sql.py: mk_flatten_mapping(): Filter str(col) through clean_name() to remove quotes, etc.

sql.py: Added clean_name()

sql.py: put_table(): Join together input tables into new table for speed and so don't modify input if values edited

sql.py: mk_flatten_mapping(): Take as_items param to return a list of dict items instead of a dict. Sort preserve cols before other cols. flatten(): Turn on as_items so that cols list is sorted in input order, with preserve cols first. This ensures that if a pkey is provided in preserve, it will be the first col in the generated table.

sql.py: mk_flatten_mapping(), flatten(): Take list of cols to select instead of using all cols in all tables to join

sql.py: mk_flatten_mapping(), flatten(): Renamed flat_table param to into to be consistent with run_query_into() and put it first because it is the output param

sql.py: Added flatten()

sql.py: mk_flatten_mapping(): preserve Col objects will have tables changed to flat_table to work with flattened table

sql.py: mk_flatten_mapping(): Added preserve param for list of columns not to rename

sql.py: esc_name_by_module(): Support module value None, and use default module psycopg2 for it

sql.py: put_table(): Renamed *pkeys_ref to *pkeys to reflect that they are now objects rather than an array-based references

sql.py: run_query_into(): Renamed into_ref param to into to reflect that it's now an object rather than an array-based reference

sql.py: run_query_into(): Made into_ref a sql_gen.Table instead of an array containing a table name to improve flexibility and clarity

dicts.py: Added join()

sql.py: Added mk_flatten_mapping()

sql.py: put_table(): Renamed the copy of in_tables that gets modified to in_tables_, so that the original list can eventually be reused in joining together the input tables into a temp table

sql.py: run_query(): FunctionValueException: Also match "date/time field value out of range" errors

sql.py: put_table(): conds: Use a set instead of a list for faster checking of the "cond not in conds" assertion

sql.py: mk_select(): conds: Support containers of any iterable type

sql.py: put_table(): Made conds a list so that there can be multiple conditions on the same column

sql.py: mk_select(): conds is list of (key, value) tuples instead of dict (dict still supported for compatibility), so that there can be multiple conditions on the same column

sql.py: mk_select(): conds is list of (key, value) tuples instead of dict (dict still supported for compatibility), so that there can be multiple conditions on the same column

util.py: NamedTuple inherits from objects.BasicObject so that it's comparable and hashable. This fixes a bug in dicts.make_hashable() where the NamedTuple created for a dict would appear to be hashable but would always compare as unequal.

sql.py: DbConn.esc_value(): Run strings.to_unicode() on the generated string so that if it contains unescaped non-ASCII characters, these will not cause problems when concatenated with plain strings

sql.py: run_query(): FunctionValueException: Unpack match.groups() into vars to make code clearer

exc.py: str_(): Avoid traceback exception-formatting functions when possible because they escape non-ASCII characters

sql.py: get_cur_query(): If no raw query: Use strings.ustr() instead of repr() to ensure that if the exception is parsed, embedded quotes will not be double-escaped. Prefix the query by [input] to show that it's not the raw query.

sql_gen.py: Non-Code objects: str() passes informative placeholder string to self.to_str() instead of empty string

sql.py: ExceptionWithNameValue: Use repr() instead of strings.ustr() on the value

sql.py: run_query(): Exception parsing: Use non-greedy qualifier "?" in regexps wherever possible to avoid matching closing quotes later in the error message