sql_gen.py: Added Array and to_Array()

sql_gen.py: Added List and inherit from it in Tuple

sql_gen.py: Renamed Tuple to Row and List to Tuple to more accurately reflect the datatype generated by each class (a Tuple being merely a grouping of values)

sql_gen.py: Moved Composite types to Literal values section as a subsection, since Composite types was really about just the input syntaxes for these types

sql_gen.py: Replaced srcs_str() with cross_join_srcs() which more correctly combines the srcs of each column using a Cartesian product. Eventually, the entire tree of srcs will need to be preserved instead of flattened in order to properly attribute errors to a specific column or set of columns.

sql_gen.py: srcs_str(): Fixed bug where needed to filter out columns with no srcs so that there aren't empty elements in the ","-separated list

sql_gen.py: Added has_srcs()

sql_gen.py: Added NestedExcHandler

sql_gen.py: Added srcs_str()

sql_gen.py: as_Col(): Support non-Code, non-string inputs by making them Literals

sql_gen.py: Added is_col() and use it in is_table_col()

sql_io.py: ExcToErrorsTable: Require users to explicitly specify an expression for the value that caused the error, instead of assuming that a variable named "value" already exists. This allows a value expression to be computed only if needed for error handling.

sql_gen.py: Moved repr() from ExcHandler to BaseExcHandler

sql_gen.py: Added BaseExcHandler and inherit from it in ExcHandlers

sql_io.py: cast(): Determining if will be saving errors: Don't add extra check if isinstance(col, sql_gen.Col) because the special case for sql_gen.Literal handles supported non-columns

sql_io.py: data_exception_handler(): Removed no longer needed db param

sql_io.py: Added ExcToErrorsTable, which separates out the errors table inserting code from the exception handling code. data_exception_handler(): Refactored to use new sql_gen.data_exception_handler() and ExcToErrorsTable.

sql_gen.py: Added data_exception_handler

sql_io.py: data_exception_handler(): Refactored to use new sql_gen.ExcToWarning when not using an errors table

sql_gen.py: Added ExcToWarning

schemas/vegbien.sql: taxondetermination: taxondetermination_taxonoccurrence_id_fkey(): Fixed bug where string containing a \-escape needed an "E" prefix

sql_io.py: data_exception_handler(): Require the caller to provide a statement to return a default value in case of error, rather than assuming the caller can accept a return value of NULL

sql_io.py: data_exception_handler(): Refactored to use new sql.define_func()

sql_io.py: put_table(): is_function: Calling function on input rows: Convert PL/Python exceptions (internal_errors) to data_exceptions using sql_gen.plpythonu_error_handler and an error handling wrapper function

debug2redmine.csv: EXPLAIN comments: Fixed bug where needed to also match whitespace at beginning of line (indent)

Use sql_gen.ReturnQuery where RETURN QUERY was previously manually prepended

sql_gen.py: Added ReturnQuery

sql.py: define_func(): Fixed bug where next_version() needed to have module name removed since it's in the same module

sql.py: mk_select(): Added explain param to turn off automatically running EXPLAIN on the created query. This is useful for SELECT statements which use local variables in PL/pgSQL functions.

sql_gen.py: with_table(): Only set the table if the passed-in value is a Col or FunctionCall

sql_gen.py: Added Tuple

sql_gen.py: Added List and use it in Values.to_str()

sql.py: Added define_func()

Use sql_gen.SetOf where SETOF was previously manually prepended

sql_gen.py: Added SetOf

sql_gen.py: FunctionDef: Support return_types which are Code objects

Use sql_gen.ColType where %TYPE was previously manually appended

sql_gen.py: Added ColType

Use sql_gen.RowType where %ROWTYPE was previously manually appended

sql_gen.py: Added RowType

sql_gen.py: RowExcIgnore: Accept row types which are Code objects

sql_gen.py: TypedCol: Accept types which are Code objects

sql_io.py: data_exception_handler(): Documented that the invalid value must be in a local variable of type text

sql_io.py: data_exception_handler(): Documented that the invalid value must be in a local variable of type text

sql_io.py: put_table(): is_function: Creating empty pkeys table so its row type can be used: Don't do this if is_literals because special error handling does not apply to that

sql_io.py: put_table(): is_function: Create empty pkeys table before calling function on all rows so its row type can later be used in an error handling wrapper function

input.Makefile: Staging tables: import/install-%: Run csv2db with a nice increment of +5 to avoid interfering with the user's other processes

root map: Run bin/map with a nice increment of +5 to avoid interfering with the user's other processes

sql_io.py: put_table(): Handle psycopg2.extensions.TransactionRollbackError by retrying the last query

sql_io.py: Creating an empty output pkeys table: Assert that there are no join columns, so that the input pkeys table will be created correctly for the empty output pkeys table

sql_io.py: put_table(): Creating an empty output pkeys table: Added "output" to clarify that the created table contains just the output pkeys, and must be joined with the input pkeys table

sql_gen.py: FunctionDef: Renamed args to params

sql_gen.py: FunctionDef: Accept parameters as FunctionParam objects instead of strings

sql_gen.py: Added FunctionParam

sql_gen.py: Added plpythonu_error_handler

Autogenerated SQL code: Use new strings.indent() where needed

strings.py: Added indent()

sql_io.py: data_exception_handler(): Refactored to use sql_gen.RowExcIgnore

sql_io.py: cast(): Refactored to use sql_gen.FunctionDef

sql_gen.py: ExcHandler: Removed extra newline after handler

sql.py: mk_insert_select(): ignore: RETURN QUERY statement: Added back missing newline after ';'

sql_gen.py: FunctionDef: Added support for parameters

sql_io.py: cast(): Just use the first word of the type in the function name to help avoid name collisions. Note that type name collisions that may be introduced by this change are not a problem because the function name is versioned. (The caching mechanism prevents versioning when the function has the same name and definition as an already-defined function.)

sql_io.py: Added data_exception_handler() and use it in cast()

sql_gen.py: ExcHandler.to_str(): Removed extra newline after body

sql_gen.py: ExcHandler: Added repr() since it's not a Code object

sql_gen.py: FunctionDef: Support custom function modifiers

sql_gen.py: RowExcIgnore: Changed exc param to exc_handler to allow user to specify handler code for the exception

sql_gen.py: Added ExcHandler, unique_violation_handler

sql_gen.py: RowExcIgnore: Don't automatically add 'RETURN QUERY' before the with_row code or ';' after it

sql_gen.py: RowExcIgnore: Allow user to specify a custom row var name

sql_gen.py: FunctionDef: Don't automatically add 'SETOF ' before the return type

sql.py: mk_insert_select(): embeddable: Use new sql_gen.RowExcIgnore

sql_gen.py: Added RowExcIgnore

sql_gen.py: FunctionDef: Determine the lang from the body's Code object instead of receiving it as a parameter

sql_gen.py: as_Code(): Fixed bug where needed to handle inputs that are already Code objects

sql_gen.py: Code: Added lang instance var

sql_gen.py: Fixed bug where Code subclasses needed to call Code.__init__() in their init() function. BasicObject: Fixed bug where init() expected a value param, when in fact the value param is something added by certain subclasses.

sql_gen.py: FunctionDef: body param: Support Code inputs in addition to strings

sql.py: mk_insert_select(): embeddable: Use new sql_gen.FunctionDef

sql_gen.py: Added FunctionDef

README.TXT: Schema changes: Documented how to reinstall errors tables

csv2db: Creating errors table: Only drop existing errors table in errors_table_only mode, so that errors tables are not unintentionally deleted when `make inputs/install` is run. This helps to make `make install` idempotent.

README.TXT: Maintenance: Full DB: Changed commands to autorotate the created backup and then test and restore a rotated backup

backups/Makefile: Added %.backup/rotate

backups/Makefile: Rearranged sections so that backup targets, which apply to both Archived imports and Full DB, are at the top of a common Backups section

inputs/import.stats.xls: Updated with stats from latest import

schemas/py_functions.sql: Disabled _date() because it does not yet output errors in a format parsable by the import process, and the import process does not yet trap errors produced by SQL functions

sql_io.py: put_table(): Determining if can use optimization for only literal values: Fixed bug where needed initial value for reduce()

sql_io.py: put_table(): Needing >= one column for INSERT SELECT: Fixed bug where can't add pkey column if calling a function instead of outputting to a table

sql_io.py: put_table(): Optimization for only literal values: Also support an empty in_tables list, for use by put()

sql_io.py: put_table(): Added optimization for only literal values, which does the same operations as put() but with the additional error handling of put_table()

pg_dump_vegbien: Don't use SET SESSION AUTHORIZATION because it doesn't work with the py_functions schema (it requires PL/Python functions to be created as user postgres and then the owner changed to bien, which SET SESSION AUTHORIZATION won't do)

sql_gen.py: Added is_literal() and use it where isinstance(..., Literal) is used

db_xml.py: put_table(): Divide fields into input columns and literal values: Translate values: Allow literal values other than strings or None (from the XML parsing), because sql_io.put_table() is getting an optimization for iterations containing only literal values, which just returns the pkey of the single row for these values (which is usually an integer) instead of a temp table with the same value in each row

bin/map: by_col: Stripping XML functions not in the DB: Remove DB functions based on whether a plain SQL function of that name exists, rather than whether a relational function (i.e. a table) of that name exists. This will allow column-based import to use plain SQL functions that don't have a corresponding relational function.

db_xml.py: Don't remove any explicit pkey because the output table may be a SQL function, which does not have a pkey. This feature only existed to support importing VegBank XML exports, which we don't use (and which would be incompatible with the schema anyway).

sql.py: function_exists(): Fixed bug where select() needed to be run with auto-rollback in case it raised an exception

xml_func.py: process(): Changed rel_funcs param to a callback is_rel_func, so that caller can specify any dynamic function to determine if a name is a relational function rather than having to list out all known relational functions

sql.py: function_exists(): Use simpler cast to regproc instead of query of information_schema.routines to determine if function exists. When the schema is not specified, this also limits the schemas checked to the search_path instead of the whole DB.