sql_gen.py: Derived.set_srcs(): Added optional overwrite param so that a default srcs value can be set only if one isn't already set

sql_gen.py: Made Table a Derived element so that a row-subset temp table could retain the schema of the table it came from, and any auxiliary tables in that schema could be located properly

sql_gen.py: Moved srcs-related functionality from Col to new superclass Derived

sql.py: cast(): save_errors: Fixed bug where srcs needed to have their names extracted before being wrapped in sql_gen.Literals. Fixed bug where errors table INSERT needed to prefix the CROSS JOIN-ed VALUES statements with SELECT * FROM because the CROSS JOIN makes it a whole SELECT query, not just a VALUES statement.

schemas/vegbien.ERD.mwb: Adjusted lines to make less intersections happen on the first page

sql.py: put_table(): MissingCastException: Fixed bug where errors_table needed to have the same schema as in_tables0 as well as part of the same name

sql_gen.py: Added suffixed_table()

sql.py: MissingCastException: Print log message that it's casting the column, to introduce the SQL function definition that follows

sql.py: put_table(): MissingCastException: Use new cast() instead of relying on existing cast functions in the database

sql.py: Added cast()

sql_gen.py: Added wrap() and use it in wrap_in_func()

sql.py: put_table(): Track the input column(s) a column is derived from, so that error messages can be attributes to the proper input column(s)

sql_gen.py: Col: Support tracking the column(s) a column is derived from, so that error messages can be attributes to the proper input column(s)

objects.py: BasicObject: Allow subclasses to customize which attrs are compared on, by adding _compare_on() method

lists.py: uniqify(): Document that it will work on any iterable, not just lists

sql.py: mk_insert_select(): embeddable: Use sql_gen.TempFunction and sql_gen.FunctionCall

sql_gen.py: Added TempFunction

schemas/functions.sql, vegbien.sql: Cast functions: Removed `RETURN new;` at end (artifact of when they were relational functions). Made the EXCEPTION block the main block of the function, to avoid unnecessary nesting.

csv2db: Errors table: index_cols: Remove no longer needed sql_gen.Col() (now done by EnsureNotNull)

sql_gen.py: EnsureNotNull: Run value through as_Col() so FunctionCall won't default it to a Literal

csv2db: Use sql_gen.EnsureNotNull instead of the ensure_not_null() function in the functions schema to avoid a dependency on the functions schema, which would cause the UNIQUE index to be dropped whenever the functions schema is reinstalled

sql_gen.py: Added EnsureNotNull

sql_gen.py: Added InternalFunction

sql_gen.py: FunctionCall: Ensure all args are Code objects using as_Value()

csv2db: Errors table: Add UNIQUE index on all columns

sql.py: add_index(): Support multiple column(s) or expression(s). Support separate table (not extracted from cols). Support UNIQUE indexes.

sql.py: add_index(): Fixed bug where expr needed to be deep copied so that any column nested in it (e.g. inside a FunctionCall) wouldn't be modified when col.table is set to None

sql.py: add_pkey(): Support multiple, custom columns

csv2db: Vacuum the created table

sql.py: Added vacuum()

sql.py: DbConn: Added with_autocommit()

csv2db: Create errors table for use by column-based import

sql.py: create_table(): Added has_pkey param to disable making the first column the primary key

csv2db: Use verbosity-based logging like bin/map. Use sql.create_table(). Add indexes on the columns to speed up column-based import and to speed up searching the table for particular values.

sql.py: create_table(): Don't add indexes on columns, because that shouldn't happen until after the table's rows have been inserted

sql.py: DbConn._db(): Output 'SET TRANSACTION ISOLATION LEVEL SERIALIZABLE' with log_level=4 because that should not be shown when the search_path is shown, which has log_level=3

sql.py: cleanup_table(): Use update(), which also fixes some formatting bugs

sql.py: DbConn._db(): Output connection configuration statements with log_level=3

sql.py: Added create_table()

sql_gen.py: Added TypedCol

sql.py: insert_select(): Pass log_level to run_query_into()

streams.py: LineCountInputStream: Fixed bug where EOF was incorrectly considered a line, causing the final line count (used by ProgressInputStream) to be off by one

sql.py: DbConn: Added print_notices() and call it after running a query

dicts.py: Added IdCompared

lists.py: Added clear()

sql.py: DbConn.do_autocommit(): Fixed typo in 'Autocommiting' debug message

sql_gen.py: ColDict: Extend dicts.DictProxy instead of UserDict.DictMixin because that already provides some of the functionality needed by ColDict

sql.py: run_query(): Only pass non-None debug_msg_ref to DbConn.run_query() if filtering with log_ignore_excs or can't mogrify() (and therefore can't print queries before they're run)

sql.py: DbConn: Added can_mogrify() and use it in mogrify()

sql.py: DbConn.run_query(): Log query before running if no debug_msg_ref specified. Documented debug_msg_ref param.

sql.py: DbConn: Added mogrify() and use it in esc_value()

schemas/functions.sql: _nullIf(): Fixed bug where wrong var name was used to retrieve type param. Reenabled _nullIf now that it's working.

sql_gen.py: ColDict.__setitem__(): Translate NULL values to the appropriate default value of the key column (which is often NULL, but not always) so that select query filtering/joins work correctly

sql.py: DbConn.col_default(): Fixed bug where returned string needed to be wrapped in sql_gen.as_Code() because it's sometimes a SQL expression and sometimes NULL

sql_gen.py: Added as_Code(). Split SQL code objects into separate sections so unparameterized classes would be separate from general classes.

sql_gen.py: Code.to_str(): Fixed bug where needed to raise NotImplementedError, not NotImplemented

dicts.py: is_dict(): Fixed bug where lists also have getitem() methods, so keys() was checked for instead

sql.py: put_table(): Fixed bug where mapping was still a plain dict because that's what dicts.join() returns, by moving the wrapping of it in a sql_gen.ColDict after dicts.join()

sql_gen.py: ColDict: Fixed bug where copy() needed to be implemented

sql.py: into_table_name(): Fixed bug where key needed to be passed through str() now that ColDict uses Col objects for everything

sql_gen.py: ColDict: Extend UserDict.DictMixin instead of dict because its non-core function implementations route all inner dict accesses to the core functions getitem() and setitem(). sql.py: put(): DuplicateKeyException: Wrap util.dict_subset_right_join() in a sql_gen.ColDict because the dict returned by util.dict_subset_right_join() is just a plain dict. (This change must happen at the same time because the previous functionality relied on a bug in ColDict.)

util.py: DefaultDict: Use dicts.DictProxy instead of collections.defaultdict so that it provides a view of the given dict instead of copying it

dicts.py: Added DictProxy

sql.py: mk_select(): conds: Support dict-like objects other than dict itself (such as sql_gen.ColDict)

dicts.py: Added is_dict()

sql.py: DbConn.DbCursor.execute(): When setting self.query, provide default values for query, params because some debug printing statements use self.query directly instead of _add_cursor_info()

sql.py: insert(): Fixed bug where column remaing needed to be removed and sql_gen.Literal value unwrapped

sql.py: mk_insert_select(): cols: Fixed bug where table name needed to be removed from columns with a table

sql_gen.py: ColDict: Only allow dict inputs to constructor so update() can later be used. Ensure keys_table is a Table object.

sql_gen.py: ColDict: Translate NULL values to the key column's actual default value. This avoids incorrectly comparing against NULL in join/filter conditions when the column's actual default value must be used.

sql.py: DbConn: Added col_default()

sql_gen.py: ColDict: Take a db param for later use in getting default column values

sql_gen.py: ColDict: Take a keys_table param for operations that require the output table. Use this to convert all string keys to sql_gen.Col objects.

sql.py: put(): Use a sql_gen.ColDict for row, just like in put_table(), so that put() will benefit from planned enhancements to sql_gen.ColDict, such as translating NULL to the default value of a NOT NULL column

schemas/functions.sql: _nullIf: Fixed bug where NOT NULL parameters were not supported, because an input NULL value would not match an existing DEFAULT value in a select query, by temporarily disabling _nullIf until this can be supported. Removed previous workarounds.

bin/map: out_is_db, row-based mode: Disabled using DB relational functions instead of XML functions because they were causing problems

sql.py: put(): DuplicateKeyException: Fixed bug where select() call needs to be surrounded by value_or_none(), not value(), because sometimes there is no entry

schemas/functions.sql: Added indexes on all relational function parameters since the unique index will not be used for lookups

schemas/py_functions.sql: Added _namePart relational function

schemas/py_functions.sql: Added _dateRangeEnd relational function

main Makefile: schemas/%/uninstall: Removed delete confirmation message because it's repetitive to keep being asked to confirm deletion when running schemas/reinstall. schemas/%/uninstall is not intended to be run by itself, so the public schema's delete confirmation message will protect the entire schemas/uninstall sequence.

schemas/py_functions.sql: _dateRangeStart(): Fixed bug where needed to get the first element of the array returned by parse_date_range()

main Makefile: VegBIEN DB: Schemas: Changed to only allow reinstalling all schemas at once (using the usual `make schemas/reinstall` command), because reinstalling just one schema would silently remove anything in other schemas that depends on that schema

schemas/py_functions.sql: Added _dateRangeStart relational function

vegbien_dest: $schemas: Added py_functions

pg_dump_vegbien: Fixed bug where needed space before $noOwners options

main Makefile: VegBIEN DB: DB and bien user: mk_db: Also `createlang plpythonu`

pg_dump_vegbien: Log the pg_dump command being run

main Makefile: VegBIEN DB: DB and bien user: Added schemas/py_functions/reset. db: Create py_functions schema.

main Makefile: VegBIEN DB: DB and bien user: Added schemas/py_functions/reset. db: Create py_functions schema.

schemas/py_functions.sql.make: Fixed bug where owners needed to be included because schema is imported as superuser so that untrusted PL/Python functions can be created

pg_dump_vegbien: Support optionally including owners

main Makefile: VegBIEN DB: DB and bien user: Factored $(confirmRm<schema>) functions message text out into $(confirmRmSchema) function

schemas/Makefile, py_functions.sql.make: Generate py_functions.sql from vegbien's py_functions schema

main Makefile: postgres-Linux: Install postgresql-plpython

main Makefile: python-Linux, postgres-Linux: Fixed bug where apt-get installs needed to each be run in a separate command, so that if any package was not found, the other packages would still install. (apt-get aborts on the first invalid package name.)

db_dump_localize: Use new pg_version

Added pg_version

sql.py: into_table_name(): If relational function has a value argument, don't include other arguments, to save space

sql.py: add_pkey(): Version the index name just in case add_suffix() doesn't correctly preserve a needed version #