sql.py: Added mk_track_data_error() and use it in cast(). This also ensures that if only one source column's row in the CROSS JOIN violates a unique constraint, other source columns' rows are still inserted.

sql_gen.py: with_default_table(): Added overwrite param to overwrite the table (if it isn't a NamedCol)

sql_gen.py: Join.to_str(): join(): Get just the table name of left_table and right_table using `.to_Table()`. Moved order switching of tables inside join() because the order reversing only applies to an individual condition.

sql_gen.py: Renamed set_default_table() to with_default_table() and copy col before modifying it so don't modify input

sql_gen.py: Added set_default_table(). as_ValueCond(): Use set_default_table() instead of as_Col() so that any name-only column also gets its table set. Join.to_str(): Parse left side using set_default_table() instead of as_Col() so that any name-only column also gets its table set.

sql_gen.py: Join: mapping param defaults to {} for e.g. CROSS JOINs. to_str(): Omit join_cond if mapping is empty, rather than if join is a specific type.

sql_gen.py: NamedValues: Change cols to Col objects with the table set to `name`

sql_gen.py: Added set_cols_table()

sql.py: mk_insert_select(): returning: Use sql_gen.to_name_only_col()

sql_gen.py: NamedTable: cols: Use sql_gen.Col objects or name strings instead of pre-rendered SQL code

sql_gen.py: NamedTable: Wrap nested code in Expr if needed

sql_gen.py: Added NamedValues

sql_gen.py: Values: Support multiple rows

sql.py: insert(): Use new sql_gen.Values

sql_gen.py: Added Values and default

sql_gen.py: Join.to_str(): Don't add join condition for CROSS JOINs

sql.py: put_table(): Factored out errors_table name setting so it can be used by ignore()

bin/map: If doing full import, clear errors table

sql.py: truncate(): Support sql_gen.Table objects

sql.py: Moved truncate() to Database structure queries section

sql.py: tables(): Run query with log_level=4 because it's a low-level structure-determining query

sql.py: table_exists(): Use new tables() exact param so that LIKE special chars in table name are not interpreted specially

sql.py: tables(): Added exact param to check for exact matches only

sql.py: put_table(): MissingCastException: Use new errors_table()

csv2db: Use new sql.errors_table()

sql.py: Added table_exists() and errors_table()

sql.py: DbConn.print_notices(): Fixed bug where it should not do anything for a MySQL connection, because that doesn't store notices the way Postgres does

sql.py: put_table(): MissingCastException: Debug message: Added Redmine formatting

schemas/functions.sql, vegbien.sql: Removed no longer needed cast functions, which are now created on the fly by column-based import

schemas/functions.sql: _nullIf(): Ignore uncastable value, because a value that's invalid for the given type is still well-defined as not matching the nullif() criterion

sql.py: put_table(): MissingCastException: Debug message: Removed "'s" so it wouldn't mess up syntax highlighting when pasting debug output into a SQL file

sql.py: cast(): Made errors table also store SQLSTATE in error_code column

sql.py: cast(): Documented that the value and error are inserted for each source column (hence the CROSS JOIN)

sql.py: cast(): Version the function name if using an errors table, to avoid collisions with other cast functions when the function name is truncated (or, more rarely, collisions with casts to the same type and on the same input columns but of a different table)

sql.py: cast(): function_name: Fixed bug where sql_gen.FunctionCall()'s parameters needed to be passed with *args syntax

sql.py: mk_flatten_mapping(): Propagate the original columns' sources to the flattened columns so they won't be lost in the flattening

sql.py: put_table(): MissingCastException: Use in_tables0's source table to locate the errors table in case in_tables0 has been subset into a temp table (which removes the schema name)

db_xml.py: put_table(): Track in_table's source so its original schema can be obtained and auxiliary tables located

sql_gen.py: Derived.set_srcs(): Added optional overwrite param so that a default srcs value can be set only if one isn't already set

sql_gen.py: Made Table a Derived element so that a row-subset temp table could retain the schema of the table it came from, and any auxiliary tables in that schema could be located properly

sql_gen.py: Moved srcs-related functionality from Col to new superclass Derived

sql.py: cast(): save_errors: Fixed bug where srcs needed to have their names extracted before being wrapped in sql_gen.Literals. Fixed bug where errors table INSERT needed to prefix the CROSS JOIN-ed VALUES statements with SELECT * FROM because the CROSS JOIN makes it a whole SELECT query, not just a VALUES statement.

schemas/vegbien.ERD.mwb: Adjusted lines to make less intersections happen on the first page

sql.py: put_table(): MissingCastException: Fixed bug where errors_table needed to have the same schema as in_tables0 as well as part of the same name

sql_gen.py: Added suffixed_table()

sql.py: MissingCastException: Print log message that it's casting the column, to introduce the SQL function definition that follows

sql.py: put_table(): MissingCastException: Use new cast() instead of relying on existing cast functions in the database

sql.py: Added cast()

sql_gen.py: Added wrap() and use it in wrap_in_func()

sql.py: put_table(): Track the input column(s) a column is derived from, so that error messages can be attributes to the proper input column(s)

sql_gen.py: Col: Support tracking the column(s) a column is derived from, so that error messages can be attributes to the proper input column(s)

objects.py: BasicObject: Allow subclasses to customize which attrs are compared on, by adding _compare_on() method

lists.py: uniqify(): Document that it will work on any iterable, not just lists

sql.py: mk_insert_select(): embeddable: Use sql_gen.TempFunction and sql_gen.FunctionCall

sql_gen.py: Added TempFunction

schemas/functions.sql, vegbien.sql: Cast functions: Removed `RETURN new;` at end (artifact of when they were relational functions). Made the EXCEPTION block the main block of the function, to avoid unnecessary nesting.

csv2db: Errors table: index_cols: Remove no longer needed sql_gen.Col() (now done by EnsureNotNull)

sql_gen.py: EnsureNotNull: Run value through as_Col() so FunctionCall won't default it to a Literal

csv2db: Use sql_gen.EnsureNotNull instead of the ensure_not_null() function in the functions schema to avoid a dependency on the functions schema, which would cause the UNIQUE index to be dropped whenever the functions schema is reinstalled

sql_gen.py: Added EnsureNotNull

sql_gen.py: Added InternalFunction

sql_gen.py: FunctionCall: Ensure all args are Code objects using as_Value()

csv2db: Errors table: Add UNIQUE index on all columns

sql.py: add_index(): Support multiple column(s) or expression(s). Support separate table (not extracted from cols). Support UNIQUE indexes.

sql.py: add_index(): Fixed bug where expr needed to be deep copied so that any column nested in it (e.g. inside a FunctionCall) wouldn't be modified when col.table is set to None

sql.py: add_pkey(): Support multiple, custom columns

csv2db: Vacuum the created table

sql.py: Added vacuum()

sql.py: DbConn: Added with_autocommit()

csv2db: Create errors table for use by column-based import

sql.py: create_table(): Added has_pkey param to disable making the first column the primary key

csv2db: Use verbosity-based logging like bin/map. Use sql.create_table(). Add indexes on the columns to speed up column-based import and to speed up searching the table for particular values.

sql.py: create_table(): Don't add indexes on columns, because that shouldn't happen until after the table's rows have been inserted

sql.py: DbConn._db(): Output 'SET TRANSACTION ISOLATION LEVEL SERIALIZABLE' with log_level=4 because that should not be shown when the search_path is shown, which has log_level=3

sql.py: cleanup_table(): Use update(), which also fixes some formatting bugs

sql.py: DbConn._db(): Output connection configuration statements with log_level=3

sql.py: Added create_table()

sql_gen.py: Added TypedCol

sql.py: insert_select(): Pass log_level to run_query_into()

streams.py: LineCountInputStream: Fixed bug where EOF was incorrectly considered a line, causing the final line count (used by ProgressInputStream) to be off by one

sql.py: DbConn: Added print_notices() and call it after running a query

dicts.py: Added IdCompared

lists.py: Added clear()

sql.py: DbConn.do_autocommit(): Fixed typo in 'Autocommiting' debug message

sql_gen.py: ColDict: Extend dicts.DictProxy instead of UserDict.DictMixin because that already provides some of the functionality needed by ColDict

sql.py: run_query(): Only pass non-None debug_msg_ref to DbConn.run_query() if filtering with log_ignore_excs or can't mogrify() (and therefore can't print queries before they're run)

sql.py: DbConn: Added can_mogrify() and use it in mogrify()

sql.py: DbConn.run_query(): Log query before running if no debug_msg_ref specified. Documented debug_msg_ref param.

sql.py: DbConn: Added mogrify() and use it in esc_value()

schemas/functions.sql: _nullIf(): Fixed bug where wrong var name was used to retrieve type param. Reenabled _nullIf now that it's working.

sql_gen.py: ColDict.__setitem__(): Translate NULL values to the appropriate default value of the key column (which is often NULL, but not always) so that select query filtering/joins work correctly

sql.py: DbConn.col_default(): Fixed bug where returned string needed to be wrapped in sql_gen.as_Code() because it's sometimes a SQL expression and sometimes NULL

sql_gen.py: Added as_Code(). Split SQL code objects into separate sections so unparameterized classes would be separate from general classes.

sql_gen.py: Code.to_str(): Fixed bug where needed to raise NotImplementedError, not NotImplemented

dicts.py: is_dict(): Fixed bug where lists also have getitem() methods, so keys() was checked for instead

sql.py: put_table(): Fixed bug where mapping was still a plain dict because that's what dicts.join() returns, by moving the wrapping of it in a sql_gen.ColDict after dicts.join()

sql_gen.py: ColDict: Fixed bug where copy() needed to be implemented

sql.py: into_table_name(): Fixed bug where key needed to be passed through str() now that ColDict uses Col objects for everything

sql_gen.py: ColDict: Extend UserDict.DictMixin instead of dict because its non-core function implementations route all inner dict accesses to the core functions getitem() and setitem(). sql.py: put(): DuplicateKeyException: Wrap util.dict_subset_right_join() in a sql_gen.ColDict because the dict returned by util.dict_subset_right_join() is just a plain dict. (This change must happen at the same time because the previous functionality relied on a bug in ColDict.)

util.py: DefaultDict: Use dicts.DictProxy instead of collections.defaultdict so that it provides a view of the given dict instead of copying it