sql.py: add_pkey(): Support multiple, custom columns

csv2db: Vacuum the created table

sql.py: Added vacuum()

sql.py: DbConn: Added with_autocommit()

csv2db: Create errors table for use by column-based import

sql.py: create_table(): Added has_pkey param to disable making the first column the primary key

csv2db: Use verbosity-based logging like bin/map. Use sql.create_table(). Add indexes on the columns to speed up column-based import and to speed up searching the table for particular values.

sql.py: create_table(): Don't add indexes on columns, because that shouldn't happen until after the table's rows have been inserted

sql.py: DbConn._db(): Output 'SET TRANSACTION ISOLATION LEVEL SERIALIZABLE' with log_level=4 because that should not be shown when the search_path is shown, which has log_level=3

sql.py: cleanup_table(): Use update(), which also fixes some formatting bugs

sql.py: DbConn._db(): Output connection configuration statements with log_level=3

sql.py: Added create_table()

sql_gen.py: Added TypedCol

sql.py: insert_select(): Pass log_level to run_query_into()

streams.py: LineCountInputStream: Fixed bug where EOF was incorrectly considered a line, causing the final line count (used by ProgressInputStream) to be off by one

sql.py: DbConn: Added print_notices() and call it after running a query

dicts.py: Added IdCompared

lists.py: Added clear()

sql.py: DbConn.do_autocommit(): Fixed typo in 'Autocommiting' debug message

sql_gen.py: ColDict: Extend dicts.DictProxy instead of UserDict.DictMixin because that already provides some of the functionality needed by ColDict

sql.py: run_query(): Only pass non-None debug_msg_ref to DbConn.run_query() if filtering with log_ignore_excs or can't mogrify() (and therefore can't print queries before they're run)

sql.py: DbConn: Added can_mogrify() and use it in mogrify()

sql.py: DbConn.run_query(): Log query before running if no debug_msg_ref specified. Documented debug_msg_ref param.

sql.py: DbConn: Added mogrify() and use it in esc_value()

schemas/functions.sql: _nullIf(): Fixed bug where wrong var name was used to retrieve type param. Reenabled _nullIf now that it's working.

sql_gen.py: ColDict.__setitem__(): Translate NULL values to the appropriate default value of the key column (which is often NULL, but not always) so that select query filtering/joins work correctly

sql.py: DbConn.col_default(): Fixed bug where returned string needed to be wrapped in sql_gen.as_Code() because it's sometimes a SQL expression and sometimes NULL

sql_gen.py: Added as_Code(). Split SQL code objects into separate sections so unparameterized classes would be separate from general classes.

sql_gen.py: Code.to_str(): Fixed bug where needed to raise NotImplementedError, not NotImplemented

dicts.py: is_dict(): Fixed bug where lists also have getitem() methods, so keys() was checked for instead

sql.py: put_table(): Fixed bug where mapping was still a plain dict because that's what dicts.join() returns, by moving the wrapping of it in a sql_gen.ColDict after dicts.join()

sql_gen.py: ColDict: Fixed bug where copy() needed to be implemented

sql.py: into_table_name(): Fixed bug where key needed to be passed through str() now that ColDict uses Col objects for everything

sql_gen.py: ColDict: Extend UserDict.DictMixin instead of dict because its non-core function implementations route all inner dict accesses to the core functions getitem() and setitem(). sql.py: put(): DuplicateKeyException: Wrap util.dict_subset_right_join() in a sql_gen.ColDict because the dict returned by util.dict_subset_right_join() is just a plain dict. (This change must happen at the same time because the previous functionality relied on a bug in ColDict.)

util.py: DefaultDict: Use dicts.DictProxy instead of collections.defaultdict so that it provides a view of the given dict instead of copying it

dicts.py: Added DictProxy

sql.py: mk_select(): conds: Support dict-like objects other than dict itself (such as sql_gen.ColDict)

dicts.py: Added is_dict()

sql.py: DbConn.DbCursor.execute(): When setting self.query, provide default values for query, params because some debug printing statements use self.query directly instead of _add_cursor_info()

sql.py: insert(): Fixed bug where column remaing needed to be removed and sql_gen.Literal value unwrapped

sql.py: mk_insert_select(): cols: Fixed bug where table name needed to be removed from columns with a table

sql_gen.py: ColDict: Only allow dict inputs to constructor so update() can later be used. Ensure keys_table is a Table object.

sql_gen.py: ColDict: Translate NULL values to the key column's actual default value. This avoids incorrectly comparing against NULL in join/filter conditions when the column's actual default value must be used.

sql.py: DbConn: Added col_default()

sql_gen.py: ColDict: Take a db param for later use in getting default column values

sql_gen.py: ColDict: Take a keys_table param for operations that require the output table. Use this to convert all string keys to sql_gen.Col objects.

sql.py: put(): Use a sql_gen.ColDict for row, just like in put_table(), so that put() will benefit from planned enhancements to sql_gen.ColDict, such as translating NULL to the default value of a NOT NULL column

schemas/functions.sql: _nullIf: Fixed bug where NOT NULL parameters were not supported, because an input NULL value would not match an existing DEFAULT value in a select query, by temporarily disabling _nullIf until this can be supported. Removed previous workarounds.

bin/map: out_is_db, row-based mode: Disabled using DB relational functions instead of XML functions because they were causing problems

sql.py: put(): DuplicateKeyException: Fixed bug where select() call needs to be surrounded by value_or_none(), not value(), because sometimes there is no entry

schemas/functions.sql: Added indexes on all relational function parameters since the unique index will not be used for lookups

schemas/py_functions.sql: Added _namePart relational function

schemas/py_functions.sql: Added _dateRangeEnd relational function

main Makefile: schemas/%/uninstall: Removed delete confirmation message because it's repetitive to keep being asked to confirm deletion when running schemas/reinstall. schemas/%/uninstall is not intended to be run by itself, so the public schema's delete confirmation message will protect the entire schemas/uninstall sequence.

schemas/py_functions.sql: _dateRangeStart(): Fixed bug where needed to get the first element of the array returned by parse_date_range()

main Makefile: VegBIEN DB: Schemas: Changed to only allow reinstalling all schemas at once (using the usual `make schemas/reinstall` command), because reinstalling just one schema would silently remove anything in other schemas that depends on that schema

schemas/py_functions.sql: Added _dateRangeStart relational function

vegbien_dest: $schemas: Added py_functions

pg_dump_vegbien: Fixed bug where needed space before $noOwners options

main Makefile: VegBIEN DB: DB and bien user: mk_db: Also `createlang plpythonu`

pg_dump_vegbien: Log the pg_dump command being run

main Makefile: VegBIEN DB: DB and bien user: Added schemas/py_functions/reset. db: Create py_functions schema.

main Makefile: VegBIEN DB: DB and bien user: Added schemas/py_functions/reset. db: Create py_functions schema.

schemas/py_functions.sql.make: Fixed bug where owners needed to be included because schema is imported as superuser so that untrusted PL/Python functions can be created

pg_dump_vegbien: Support optionally including owners

main Makefile: VegBIEN DB: DB and bien user: Factored $(confirmRm<schema>) functions message text out into $(confirmRmSchema) function

schemas/Makefile, py_functions.sql.make: Generate py_functions.sql from vegbien's py_functions schema

main Makefile: postgres-Linux: Install postgresql-plpython

main Makefile: python-Linux, postgres-Linux: Fixed bug where apt-get installs needed to each be run in a separate command, so that if any package was not found, the other packages would still install. (apt-get aborts on the first invalid package name.)

db_dump_localize: Use new pg_version

Added pg_version

sql.py: into_table_name(): If relational function has a value argument, don't include other arguments, to save space

sql.py: add_pkey(): Version the index name just in case add_suffix() doesn't correctly preserve a needed version #

sql_gen.py: add_suffix(): Fixed bug where only strings already at the max length had the version preserved, even though appending the suffix could bring it past the max length and still cause the version to be overwritten. Fixed bug where last # in str, not first, should be considered to precede the version.

sql.py: put_table(): mapping param: Fixed documentation of supported key/value types

db_xml.py: put_table(): Removed no longer accurate comment about handling _simplifyPath

schemas/functions.sql: Added _nullIf relational function

sql_gen.py: add_suffix(): Preserve version so that it won't be truncated off the string, leading to collisions

sql_gen.py: identifier_max_len: Fixed bug where PostgreSQL's max length was actually 63, not 64

schemas/functions.sql: _label(): Fixed bug where some Python syntax had not been translated to PostgreSQL

schemas/functions.sql: Added _label relational function

db_xml.py: put_table(): Subsetting in_table: Fixed bug where in_table was not being ordered by the row_num, because order_by was set to None when it should have been omitted so it would default to the pkey

csv2db: Increased frequency of "Processed .. row(s)" messages to match slower, more common INSERT case instead of faster, less used COPY FROM case

schemas/functions.sql: _merge(): Fixed bug where values were ordered by value instead of by sort order (column name)

xml_func.py: process(): Refactored to emphasize special handling for row-based and column-based modes. In row-based mode, always use a DB relational function over a local XML function when possible, to faciliate testing of DB relational functions in row-based mode. (The shadowed local XML version will still be tested in non-DB modes, such as outputting to intermediate XML files.)

bin/map: Move retrieval of out_db's relational functions outside of process_input() so they can also be used by the non-by_col case

bin/map: out_is_db: Don't evaluate relational functions in xml_func.process() because these will be evaluated by db_xml.put()

xml_func.py: Removed no longer used strip()

bin/map: Use xml_func.process(..., strip=True) instead of xml_func.strip()

xml_func.py: process(): Added strip()'s functionality via strip option

schemas/functions.sql: Added _merge relational function

schemas/functions.sql: Added join_strs() aggregate

sql.py: Renamed index_pkey() to add_pkey() to be consistent with add_index()

sql.py: into_table_name(): In function args, omit column name for function result columns

sql.py: into_table_name(): In function args, keep the input table name for input columns to identify where they came from, except for the main input table name because it makes the string too long

sql_gen.py: esc_name(): Don't return plain name if is_safe_name(), because this makes the SQL inconsistent when some names have "_"s and some don't

sql.py: index_pkey(): Use sql_gen.add_suffix() to ensure index name isn't too long

sql.py: put_table(): insert_out_pkeys, insert_in_pkeys: Use sql_gen.add_suffix() to ensure name isn't too long

sql.py: next_version(): Use new sql_gen.add_suffix(). Removed identifier_max_len because it is now in sql_gen.

sql_gen.py: Added identifier_max_len and add_suffix()