sql_gen.py: ColDict: Only allow dict inputs to constructor so update() can later be used. Ensure keys_table is a Table object.

sql_gen.py: ColDict: Translate NULL values to the key column's actual default value. This avoids incorrectly comparing against NULL in join/filter conditions when the column's actual default value must be used.

sql.py: DbConn: Added col_default()

sql_gen.py: ColDict: Take a db param for later use in getting default column values

sql_gen.py: ColDict: Take a keys_table param for operations that require the output table. Use this to convert all string keys to sql_gen.Col objects.

sql.py: put(): Use a sql_gen.ColDict for row, just like in put_table(), so that put() will benefit from planned enhancements to sql_gen.ColDict, such as translating NULL to the default value of a NOT NULL column

schemas/functions.sql: _nullIf: Fixed bug where NOT NULL parameters were not supported, because an input NULL value would not match an existing DEFAULT value in a select query, by temporarily disabling _nullIf until this can be supported. Removed previous workarounds.

bin/map: out_is_db, row-based mode: Disabled using DB relational functions instead of XML functions because they were causing problems

sql.py: put(): DuplicateKeyException: Fixed bug where select() call needs to be surrounded by value_or_none(), not value(), because sometimes there is no entry

schemas/functions.sql: Added indexes on all relational function parameters since the unique index will not be used for lookups

schemas/py_functions.sql: Added _namePart relational function

schemas/py_functions.sql: Added _dateRangeEnd relational function

main Makefile: schemas/%/uninstall: Removed delete confirmation message because it's repetitive to keep being asked to confirm deletion when running schemas/reinstall. schemas/%/uninstall is not intended to be run by itself, so the public schema's delete confirmation message will protect the entire schemas/uninstall sequence.

schemas/py_functions.sql: _dateRangeStart(): Fixed bug where needed to get the first element of the array returned by parse_date_range()

main Makefile: VegBIEN DB: Schemas: Changed to only allow reinstalling all schemas at once (using the usual `make schemas/reinstall` command), because reinstalling just one schema would silently remove anything in other schemas that depends on that schema

schemas/py_functions.sql: Added _dateRangeStart relational function

vegbien_dest: $schemas: Added py_functions

pg_dump_vegbien: Fixed bug where needed space before $noOwners options

main Makefile: VegBIEN DB: DB and bien user: mk_db: Also `createlang plpythonu`

pg_dump_vegbien: Log the pg_dump command being run

main Makefile: VegBIEN DB: DB and bien user: Added schemas/py_functions/reset. db: Create py_functions schema.

main Makefile: VegBIEN DB: DB and bien user: Added schemas/py_functions/reset. db: Create py_functions schema.

schemas/py_functions.sql.make: Fixed bug where owners needed to be included because schema is imported as superuser so that untrusted PL/Python functions can be created

pg_dump_vegbien: Support optionally including owners

main Makefile: VegBIEN DB: DB and bien user: Factored $(confirmRm<schema>) functions message text out into $(confirmRmSchema) function

schemas/Makefile, py_functions.sql.make: Generate py_functions.sql from vegbien's py_functions schema

main Makefile: postgres-Linux: Install postgresql-plpython

main Makefile: python-Linux, postgres-Linux: Fixed bug where apt-get installs needed to each be run in a separate command, so that if any package was not found, the other packages would still install. (apt-get aborts on the first invalid package name.)

db_dump_localize: Use new pg_version

Added pg_version

sql.py: into_table_name(): If relational function has a value argument, don't include other arguments, to save space

sql.py: add_pkey(): Version the index name just in case add_suffix() doesn't correctly preserve a needed version #

sql_gen.py: add_suffix(): Fixed bug where only strings already at the max length had the version preserved, even though appending the suffix could bring it past the max length and still cause the version to be overwritten. Fixed bug where last # in str, not first, should be considered to precede the version.

sql.py: put_table(): mapping param: Fixed documentation of supported key/value types

db_xml.py: put_table(): Removed no longer accurate comment about handling _simplifyPath

schemas/functions.sql: Added _nullIf relational function

sql_gen.py: add_suffix(): Preserve version so that it won't be truncated off the string, leading to collisions

sql_gen.py: identifier_max_len: Fixed bug where PostgreSQL's max length was actually 63, not 64

schemas/functions.sql: _label(): Fixed bug where some Python syntax had not been translated to PostgreSQL

schemas/functions.sql: Added _label relational function

db_xml.py: put_table(): Subsetting in_table: Fixed bug where in_table was not being ordered by the row_num, because order_by was set to None when it should have been omitted so it would default to the pkey

csv2db: Increased frequency of "Processed .. row(s)" messages to match slower, more common INSERT case instead of faster, less used COPY FROM case

schemas/functions.sql: _merge(): Fixed bug where values were ordered by value instead of by sort order (column name)

xml_func.py: process(): Refactored to emphasize special handling for row-based and column-based modes. In row-based mode, always use a DB relational function over a local XML function when possible, to faciliate testing of DB relational functions in row-based mode. (The shadowed local XML version will still be tested in non-DB modes, such as outputting to intermediate XML files.)

bin/map: Move retrieval of out_db's relational functions outside of process_input() so they can also be used by the non-by_col case

bin/map: out_is_db: Don't evaluate relational functions in xml_func.process() because these will be evaluated by db_xml.put()

xml_func.py: Removed no longer used strip()

bin/map: Use xml_func.process(..., strip=True) instead of xml_func.strip()

xml_func.py: process(): Added strip()'s functionality via strip option

schemas/functions.sql: Added _merge relational function

schemas/functions.sql: Added join_strs() aggregate

sql.py: Renamed index_pkey() to add_pkey() to be consistent with add_index()

sql.py: into_table_name(): In function args, omit column name for function result columns

sql.py: into_table_name(): In function args, keep the input table name for input columns to identify where they came from, except for the main input table name because it makes the string too long

sql_gen.py: esc_name(): Don't return plain name if is_safe_name(), because this makes the SQL inconsistent when some names have "_"s and some don't

sql.py: index_pkey(): Use sql_gen.add_suffix() to ensure index name isn't too long

sql.py: put_table(): insert_out_pkeys, insert_in_pkeys: Use sql_gen.add_suffix() to ensure name isn't too long

sql.py: next_version(): Use new sql_gen.add_suffix(). Removed identifier_max_len because it is now in sql_gen.

sql_gen.py: Added identifier_max_len and add_suffix()

next_version(): Append the version # so it looks more natural. Take into account the max identifier length.

strings.py: Added add_suffix()

sql.py: put_table(): Name the in_table just "in" plus the version #, and the insert_in_pkeys/insert_out_pkeys based on in_table, so that they don't take up so much space in the SQL

sql_gen.py: is_safe_name(): Fixed bug where keywords were incorrectly considered safe

strings.py: repr_no_u(): Fixed bug where "u" prefix was removed even in reprs of non-strings

db_xml.py: into_table_name(): Removed no longer necessary handling of simple functions, which is now done by sql.into_table_name(). Ensure that rank params in functions (not tables) are not treated specially as hierarchical.

sql.py: put_table(): If into == None: For function calls, include the arguments in the into table name

sql_gen.py: to_name_only_col(): Support non-Col Code inputs

sql_gen.py CompareCond.to_str(), callers of combine_conds(): Removed unnecessary grouping () to make SQL clearer

sql_gen.py: Added combine_conds() and use it in Join.to_str() and sql.py mk_select()

sql_gen.py Join.to_str(), sql.py mk_select(): Combining conditions: Don't add newlines where not needed, so that output is less vertically spread out

sql_gen.py: is_safe_name(): Fixed bug where names starting with a digit were incorrectly considered safe

sql.py: put_table(): Separate temp table names from into table name with "_" instead of "-" so that quoting the table name will usually be unnecessary

sql.py: esc_name_by_module(): Remove unused param ignore_case

sql_gen.py: esc_name(): If is_safe_name(), just return name, to avoid escessive escaping in debug output for Redmine

sql_gen.py: is_safe_name(): Don't consider uppercase letters safe because they would cause inconsistent behavior in PostgreSQL if quoted vs. not quoted (only unquoted identifiers are case-insensitive)

sql.py: Removed no longer needed check_name()

sql.py: esc_name_by_module(): psycopg2: If ignore_case is set but name is unsafe, just escape it instead of raising an exception

sql_gen.py: Added is_safe_name()

sql.py: put_table(): col_ustr(): Removed no longer needed sql_gen.as_Col() because mapping and join_cols now ensure that their contents are sql_gen.Col objects

schemas/functions.sql: Added _alt relational function

sql.py: put_table(): Make mapping and join_cols a sql_gen.ColDict so that literal values will always be turned into sql_gen.Col objects. DuplicateKeyException: Use dict_subset_right_join() instead of dict_subset() so that all columns in a constraint are included in joins on out_table (such as for a relational function with omitted arguments).

sql_gen.py: Added ColDict

sql_gen.py: as_Col(): Added optional name param to specify that non-Col input will be renamed using NamedCol with the given name

sql.py: put_table(): FunctionValueException: Fixed bug where only function calls, not plain columns, were handled, by using sql_gen.unwrap_func_call() to remove any function call only if there was one

sql_gen.py: Added unwrap_func_call()

bin/map: by_col: Stripping XML functions not in the DB: Fixed bug where preserve_funcs.add() was used when `preserve_funcs |=` should have been used to add the entire iterable that sql.tables() returns

sql.py: not_null_col: Changed value to 'not_null_col' so that column doesn't seem like a status indicator of whether some value is not null (in fact it's just a column that is always not null)

xml_func.py: Replaced xpath.get_1() with xpath.get_value() where possible, for simplicity

xml_func.py: strip(): Evaluate structural functions like _ignore and _ref by process() instead of removing them. Store structural functions' names in structural_funcs module var. This ensures that _ref targets are still expanded in column-based import.

xpath.py: get(): Create attrs: Put keys last so that any lookahead assertion's path will be created last as it would have without the assertion. This ensures that any value argument of an XML function will always go last even if a lookahead assertion would otherwise have caused it to be created with the element's keys, which previously were created before the attributes.

sql.py: put_table(): If is_func, default into table name ends in () instead of '-pkeys'

schemas/vegbien.sql, functions.sql: Made cast functions STRICT to enable the RETURNS NULL ON NULL INPUT optimization

db_xml.py: put_table(): Pass is_func to sql.put_table()

sql.py: put_table(): Added is_func param for whether out_table is the name of a SQL function, not a table

db_xml.py: put_table(): Treat every node name that starts with "_" as a function, not just members of put_table_special_funcs. This ensures that DB function args are always treated as values, not children with fkeys to parent.

bin/map: by_col: Strip only XML functions that are not in the DB

db_xml.py: put_table(): Make special_funcs externally available as module constant put_table_special_funcs

sql.py: tables(): Changed schema param to schema_like and filter the schema using LIKE so that all schemas can be selected

to_do/timeline.doc: Updated to reflect the month we spent on optimization and column-based import

sql.py: put_table(): in_table name: Remove '-pkeys' suffix from the into table name before adding '-input' so that the name is shorter and clearer