Activity
From 05/10/2012 to 06/08/2012
06/08/2012
- 09:59 PM Revision 2725: sql.py: DbConn.print_notices(): Fixed bug where it should not do anything for a MySQL connection, because that doesn't store notices the way Postgres does
- 09:54 PM Revision 2724: sql.py: put_table(): MissingCastException: Debug message: Added Redmine formatting
- 09:48 PM Revision 2723: schemas/functions.sql, vegbien.sql: Removed no longer needed cast functions, which are now created on the fly by column-based import
- 09:45 PM Revision 2722: schemas/functions.sql: _nullIf(): Ignore uncastable value, because a value that's invalid for the given type is still well-defined as not matching the nullif() criterion
- 09:25 PM Revision 2721: sql.py: put_table(): MissingCastException: Debug message: Removed "'s" so it wouldn't mess up syntax highlighting when pasting debug output into a SQL file
- 09:08 PM Revision 2720: sql.py: cast(): Made errors table also store SQLSTATE in error_code column
- 08:40 PM Revision 2719: sql.py: cast(): Documented that the value and error are inserted for *each* source column (hence the CROSS JOIN)
- 08:39 PM Revision 2718: sql.py: cast(): Version the function name if using an errors table, to avoid collisions with other cast functions when the function name is truncated (or, more rarely, collisions with casts to the same type and on the same input columns but of a different table)
- 08:25 PM Revision 2717: sql.py: cast(): function_name: Fixed bug where sql_gen.FunctionCall()'s parameters needed to be passed with *args syntax
- 08:23 PM Revision 2716: sql.py: mk_flatten_mapping(): Propagate the original columns' sources to the flattened columns so they won't be lost in the flattening
- 08:20 PM Revision 2715: sql.py: put_table(): MissingCastException: Use in_tables0's source table to locate the errors table in case in_tables0 has been subset into a temp table (which removes the schema name)
- 08:19 PM Revision 2714: db_xml.py: put_table(): Track in_table's source so its original schema can be obtained and auxiliary tables located
- 08:17 PM Revision 2713: sql_gen.py: Derived.set_srcs(): Added optional overwrite param so that a default srcs value can be set only if one isn't already set
- 08:04 PM Revision 2712: sql_gen.py: Made Table a Derived element so that a row-subset temp table could retain the schema of the table it came from, and any auxiliary tables in that schema could be located properly
- 08:01 PM Revision 2711: sql_gen.py: Moved srcs-related functionality from Col to new superclass Derived
- 07:53 PM Revision 2710: sql.py: cast(): save_errors: Fixed bug where srcs needed to have their names extracted before being wrapped in sql_gen.Literals. Fixed bug where errors table INSERT needed to prefix the CROSS JOIN-ed VALUES statements with SELECT * FROM because the CROSS JOIN makes it a whole SELECT query, not just a VALUES statement.
- 07:40 PM Revision 2709: schemas/vegbien.ERD.mwb: Adjusted lines to make less intersections happen on the first page
- 07:27 PM Revision 2708: sql.py: put_table(): MissingCastException: Fixed bug where errors_table needed to have the same schema as in_tables0 as well as part of the same name
- 07:23 PM Revision 2707: sql_gen.py: Added suffixed_table()
- 07:14 PM Revision 2706: sql.py: MissingCastException: Print log message that it's casting the column, to introduce the SQL function definition that follows
- 07:06 PM Revision 2705: sql.py: put_table(): MissingCastException: Use new cast() instead of relying on existing cast functions in the database
- 07:04 PM Revision 2704: sql.py: Added cast()
- 06:00 PM Revision 2703: sql_gen.py: Added wrap() and use it in wrap_in_func()
- 05:36 PM Revision 2702: sql.py: put_table(): Track the input column(s) a column is derived from, so that error messages can be attributes to the proper input column(s)
- 05:35 PM Revision 2701: sql_gen.py: Col: Support tracking the column(s) a column is derived from, so that error messages can be attributes to the proper input column(s)
- 05:33 PM Revision 2700: objects.py: BasicObject: Allow subclasses to customize which attrs are compared on, by adding _compare_on() method
- 04:45 PM Revision 2699: lists.py: uniqify(): Document that it will work on any iterable, not just lists
- 04:06 PM Revision 2698: sql.py: mk_insert_select(): embeddable: Use sql_gen.TempFunction and sql_gen.FunctionCall
- 04:04 PM Revision 2697: sql_gen.py: Added TempFunction
- 03:26 PM Revision 2696: schemas/functions.sql, vegbien.sql: Cast functions: Removed `RETURN new;` at end (artifact of when they were relational functions). Made the EXCEPTION block the main block of the function, to avoid unnecessary nesting.
- 02:26 PM Revision 2695: csv2db: Errors table: index_cols: Remove no longer needed sql_gen.Col() (now done by EnsureNotNull)
- 02:25 PM Revision 2694: sql_gen.py: EnsureNotNull: Run value through as_Col() so FunctionCall won't default it to a Literal
- 02:19 PM Revision 2693: csv2db: Use sql_gen.EnsureNotNull instead of the ensure_not_null() function in the functions schema to avoid a dependency on the functions schema, which would cause the UNIQUE index to be dropped whenever the functions schema is reinstalled
- 02:18 PM Revision 2692: sql_gen.py: Added EnsureNotNull
- 02:17 PM Revision 2691: sql_gen.py: Added InternalFunction
- 02:08 PM Revision 2690: sql_gen.py: FunctionCall: Ensure all args are Code objects using as_Value()
- 01:51 PM Revision 2689: csv2db: Errors table: Add UNIQUE index on all columns
- 01:50 PM Revision 2688: sql.py: add_index(): Support multiple column(s) or expression(s). Support separate table (not extracted from cols). Support UNIQUE indexes.
06/07/2012
- 09:41 PM Revision 2687: sql.py: add_index(): Fixed bug where expr needed to be deep copied so that any column nested in it (e.g. inside a FunctionCall) wouldn't be modified when col.table is set to None
- 09:35 PM Revision 2686: sql.py: add_pkey(): Support multiple, custom columns
- 09:24 PM Revision 2685: csv2db: Vacuum the created table
- 09:23 PM Revision 2684: sql.py: Added vacuum()
- 09:23 PM Revision 2683: sql.py: DbConn: Added with_autocommit()
- 08:58 PM Revision 2682: csv2db: Create errors table for use by column-based import
- 08:57 PM Revision 2681: sql.py: create_table(): Added has_pkey param to disable making the first column the primary key
- 08:21 PM Revision 2680: csv2db: Use verbosity-based logging like bin/map. Use sql.create_table(). Add indexes on the columns to speed up column-based import and to speed up searching the table for particular values.
- 08:00 PM Revision 2679: sql.py: create_table(): Don't add indexes on columns, because that shouldn't happen until after the table's rows have been inserted
- 07:55 PM Revision 2678: sql.py: DbConn._db(): Output 'SET TRANSACTION ISOLATION LEVEL SERIALIZABLE' with log_level=4 because that should not be shown when the search_path is shown, which has log_level=3
- 07:43 PM Revision 2677: sql.py: cleanup_table(): Use update(), which also fixes some formatting bugs
- 07:24 PM Revision 2676: sql.py: DbConn._db(): Output connection configuration statements with log_level=3
- 07:15 PM Revision 2675: sql.py: Added create_table()
- 07:14 PM Revision 2674: sql_gen.py: Added TypedCol
- 06:59 PM Revision 2673: sql.py: insert_select(): Pass log_level to run_query_into()
- 05:15 PM Revision 2672: streams.py: LineCountInputStream: Fixed bug where EOF was incorrectly considered a line, causing the final line count (used by ProgressInputStream) to be off by one
- 04:25 PM Revision 2671: sql.py: DbConn: Added print_notices() and call it after running a query
- 04:12 PM Revision 2670: dicts.py: Added IdCompared
- 03:50 PM Revision 2669: lists.py: Added clear()
- 03:24 PM Revision 2668: sql.py: DbConn.do_autocommit(): Fixed typo in 'Autocommiting' debug message
- 03:20 PM Revision 2667: sql_gen.py: ColDict: Extend dicts.DictProxy instead of UserDict.DictMixin because that already provides some of the functionality needed by ColDict
06/06/2012
- 09:46 PM Revision 2666: sql.py: run_query(): Only pass non-None debug_msg_ref to DbConn.run_query() if filtering with log_ignore_excs or can't mogrify() (and therefore can't print queries before they're run)
- 09:36 PM Revision 2665: sql.py: DbConn: Added can_mogrify() and use it in mogrify()
- 09:35 PM Revision 2664: sql.py: DbConn.run_query(): Log query before running if no debug_msg_ref specified. Documented debug_msg_ref param.
- 08:43 PM Revision 2663: sql.py: DbConn: Added mogrify() and use it in esc_value()
- 07:58 PM Revision 2662: schemas/functions.sql: _nullIf(): Fixed bug where wrong var name was used to retrieve type param. Reenabled _nullIf now that it's working.
- 07:25 PM Revision 2661: sql_gen.py: ColDict.__setitem__(): Translate NULL values to the appropriate default value of the key column (which is often NULL, but not always) so that select query filtering/joins work correctly
- 07:21 PM Revision 2660: sql.py: DbConn.col_default(): Fixed bug where returned string needed to be wrapped in sql_gen.as_Code() because it's sometimes a SQL expression and sometimes NULL
- 07:20 PM Revision 2659: sql_gen.py: Added as_Code(). Split SQL code objects into separate sections so unparameterized classes would be separate from general classes.
- 07:08 PM Revision 2658: sql_gen.py: Code.to_str(): Fixed bug where needed to raise NotImplementedError, not NotImplemented
- 06:58 PM Revision 2657: dicts.py: is_dict(): Fixed bug where lists also have __getitem__() methods, so keys() was checked for instead
- 06:44 PM Revision 2656: sql.py: put_table(): Fixed bug where mapping was still a plain dict because that's what dicts.join() returns, by moving the wrapping of it in a sql_gen.ColDict after dicts.join()
- 06:28 PM Revision 2655: sql_gen.py: ColDict: Fixed bug where copy() needed to be implemented
- 06:14 PM Revision 2654: sql.py: into_table_name(): Fixed bug where key needed to be passed through str() now that ColDict uses Col objects for everything
- 06:11 PM Revision 2653: sql_gen.py: ColDict: Extend UserDict.DictMixin instead of dict because its non-core function implementations route all inner dict accesses to the core functions __getitem__() and __setitem__(). sql.py: put(): DuplicateKeyException: Wrap util.dict_subset_right_join() in a sql_gen.ColDict because the dict returned by util.dict_subset_right_join() is just a plain dict. (This change must happen at the same time because the previous functionality relied on a bug in ColDict.)
- 06:04 PM Revision 2652: util.py: DefaultDict: Use dicts.DictProxy instead of collections.defaultdict so that it provides a view of the given dict instead of copying it
- 06:03 PM Revision 2651: dicts.py: Added DictProxy
- 05:37 PM Revision 2650: sql.py: mk_select(): conds: Support dict-like objects other than dict itself (such as sql_gen.ColDict)
- 05:16 PM Revision 2649: dicts.py: Added is_dict()
- 05:06 PM Revision 2648: sql.py: DbConn.DbCursor.execute(): When setting self.query, provide default values for query, params because some debug printing statements use self.query directly instead of _add_cursor_info()
- 05:01 PM Revision 2647: sql.py: insert(): Fixed bug where column remaing needed to be removed and sql_gen.Literal value unwrapped
- 03:40 PM Revision 2646: sql.py: mk_insert_select(): cols: Fixed bug where table name needed to be removed from columns with a table
- 03:40 PM Revision 2645: sql_gen.py: ColDict: Only allow dict inputs to constructor so update() can later be used. Ensure keys_table is a Table object.
- 02:55 PM Revision 2644: sql_gen.py: ColDict: Translate NULL values to the key column's actual default value. This avoids incorrectly comparing against NULL in join/filter conditions when the column's actual default value must be used.
- 02:52 PM Revision 2643: sql.py: DbConn: Added col_default()
- 02:52 PM Revision 2642: sql_gen.py: ColDict: Take a db param for later use in getting default column values
- 02:16 PM Revision 2641: sql_gen.py: ColDict: Take a keys_table param for operations that require the output table. Use this to convert all string keys to sql_gen.Col objects.
- 01:55 PM Revision 2640: sql.py: put(): Use a sql_gen.ColDict for row, just like in put_table(), so that put() will benefit from planned enhancements to sql_gen.ColDict, such as translating NULL to the default value of a NOT NULL column
- 01:38 PM Revision 2639: schemas/functions.sql: _nullIf: Fixed bug where NOT NULL parameters were not supported, because an input NULL value would not match an existing DEFAULT value in a select query, by temporarily disabling _nullIf until this can be supported. Removed previous workarounds.
06/05/2012
- 03:21 PM Revision 2638: bin/map: out_is_db, row-based mode: Disabled using DB relational functions instead of XML functions because they were causing problems
- 03:18 PM Revision 2637: sql.py: put(): DuplicateKeyException: Fixed bug where select() call needs to be surrounded by value_or_none(), not value(), because sometimes there is no entry
- 03:17 PM Revision 2636: schemas/functions.sql: Added indexes on all relational function parameters since the unique index will not be used for lookups
- 02:29 PM Revision 2635: schemas/py_functions.sql: Added _namePart relational function
- 01:06 PM Revision 2634: schemas/py_functions.sql: Added _dateRangeEnd relational function
- 12:59 PM Revision 2633: main Makefile: schemas/%/uninstall: Removed delete confirmation message because it's repetitive to keep being asked to confirm deletion when running schemas/reinstall. schemas/%/uninstall is not intended to be run by itself, so the public schema's delete confirmation message will protect the entire schemas/uninstall sequence.
- 12:55 PM Revision 2632: schemas/py_functions.sql: _dateRangeStart(): Fixed bug where needed to get the first element of the array returned by parse_date_range()
- 12:32 PM Revision 2631: main Makefile: VegBIEN DB: Schemas: Changed to only allow reinstalling all schemas at once (using the usual `make schemas/reinstall` command), because reinstalling just one schema would silently remove anything in other schemas that depends on that schema
- 11:52 AM Revision 2630: schemas/py_functions.sql: Added _dateRangeStart relational function
- 10:53 AM Revision 2629: vegbien_dest: $schemas: Added py_functions
- 10:49 AM Revision 2628: pg_dump_vegbien: Fixed bug where needed space before $noOwners options
- 10:37 AM Revision 2627: main Makefile: VegBIEN DB: DB and bien user: mk_db: Also `createlang plpythonu`
- 10:26 AM Revision 2626: pg_dump_vegbien: Log the pg_dump command being run
- 10:23 AM Revision 2625: main Makefile: VegBIEN DB: DB and bien user: Added schemas/py_functions/reset. db: Create py_functions schema.
- 10:22 AM Revision 2624: main Makefile: VegBIEN DB: DB and bien user: Added schemas/py_functions/reset. db: Create py_functions schema.
- 10:16 AM Revision 2623: schemas/py_functions.sql.make: Fixed bug where owners needed to be included because schema is imported as superuser so that untrusted PL/Python functions can be created
- 10:15 AM Revision 2622: pg_dump_vegbien: Support optionally including owners
- 09:59 AM Revision 2621: main Makefile: VegBIEN DB: DB and bien user: Factored $(confirmRm<schema>) functions message text out into $(confirmRmSchema) function
- 09:52 AM Revision 2620: schemas/Makefile, py_functions.sql.make: Generate py_functions.sql from vegbien's py_functions schema
- 09:32 AM Revision 2619: main Makefile: postgres-Linux: Install postgresql-plpython
- 09:27 AM Revision 2618: main Makefile: python-Linux, postgres-Linux: Fixed bug where apt-get installs needed to each be run in a separate command, so that if any package was not found, the other packages would still install. (apt-get aborts on the first invalid package name.)
- 09:18 AM Revision 2617: db_dump_localize: Use new pg_version
- 09:18 AM Revision 2616: Added pg_version
- 08:05 AM Revision 2615: sql.py: into_table_name(): If relational function has a value argument, don't include other arguments, to save space
- 08:03 AM Revision 2614: sql.py: add_pkey(): Version the index name just in case add_suffix() doesn't correctly preserve a needed version #
- 08:01 AM Revision 2613: sql_gen.py: add_suffix(): Fixed bug where only strings already at the max length had the version preserved, even though appending the suffix could bring it past the max length and still cause the version to be overwritten. Fixed bug where last # in str, not first, should be considered to precede the version.
- 07:46 AM Revision 2612: sql.py: put_table(): mapping param: Fixed documentation of supported key/value types
- 07:09 AM Revision 2611: db_xml.py: put_table(): Removed no longer accurate comment about handling _simplifyPath
- 07:01 AM Revision 2610: schemas/functions.sql: Added _nullIf relational function
- 06:39 AM Revision 2609: sql_gen.py: add_suffix(): Preserve version so that it won't be truncated off the string, leading to collisions
06/04/2012
- 03:35 PM Revision 2608: sql_gen.py: identifier_max_len: Fixed bug where PostgreSQL's max length was actually 63, not 64
- 03:18 PM Revision 2607: schemas/functions.sql: _label(): Fixed bug where some Python syntax had not been translated to PostgreSQL
- 03:07 PM Revision 2606: schemas/functions.sql: Added _label relational function
- 03:06 PM Revision 2605: db_xml.py: put_table(): Subsetting in_table: Fixed bug where in_table was not being ordered by the row_num, because order_by was set to None when it should have been omitted so it would default to the pkey
- 02:51 PM Revision 2604: csv2db: Increased frequency of "Processed .. row(s)" messages to match slower, more common INSERT case instead of faster, less used COPY FROM case
- 02:40 PM Revision 2603: schemas/functions.sql: _merge(): Fixed bug where values were ordered by value instead of by sort order (column name)
- 02:17 PM Revision 2602: xml_func.py: process(): Refactored to emphasize special handling for row-based and column-based modes. In row-based mode, always use a DB relational function over a local XML function when possible, to faciliate testing of DB relational functions in row-based mode. (The shadowed local XML version will still be tested in non-DB modes, such as outputting to intermediate XML files.)
- 01:01 PM Revision 2601: bin/map: Move retrieval of out_db's relational functions outside of process_input() so they can also be used by the non-by_col case
- 12:52 PM Revision 2600: bin/map: out_is_db: Don't evaluate relational functions in xml_func.process() because these will be evaluated by db_xml.put()
- 12:41 PM Revision 2599: xml_func.py: Removed no longer used strip()
- 12:40 PM Revision 2598: bin/map: Use xml_func.process(..., strip=True) instead of xml_func.strip()
- 12:39 PM Revision 2597: xml_func.py: process(): Added strip()'s functionality via strip option
- 12:10 PM Revision 2596: schemas/functions.sql: Added _merge relational function
- 11:48 AM Revision 2595: schemas/functions.sql: Added join_strs() aggregate
- 10:21 AM Revision 2594: sql.py: Renamed index_pkey() to add_pkey() to be consistent with add_index()
- 10:07 AM Revision 2593: sql.py: into_table_name(): In function args, omit column name for function result columns
- 09:57 AM Revision 2592: sql.py: into_table_name(): In function args, keep the input table name for input columns to identify where they came from, except for the *main* input table name because it makes the string too long
- 09:22 AM Revision 2591: sql_gen.py: esc_name(): Don't return plain name if is_safe_name(), because this makes the SQL inconsistent when some names have "_"s and some don't
- 09:17 AM Revision 2590: sql.py: index_pkey(): Use sql_gen.add_suffix() to ensure index name isn't too long
- 09:15 AM Revision 2589: sql.py: put_table(): insert_out_pkeys, insert_in_pkeys: Use sql_gen.add_suffix() to ensure name isn't too long
- 09:07 AM Revision 2588: sql.py: next_version(): Use new sql_gen.add_suffix(). Removed identifier_max_len because it is now in sql_gen.
- 09:07 AM Revision 2587: sql_gen.py: Added identifier_max_len and add_suffix()
- 09:04 AM Revision 2586: next_version(): Append the version # so it looks more natural. Take into account the max identifier length.
- 09:03 AM Revision 2585: strings.py: Added add_suffix()
- 08:51 AM Revision 2584: sql.py: put_table(): Name the in_table just "in" plus the version #, and the insert_in_pkeys/insert_out_pkeys based on in_table, so that they don't take up so much space in the SQL
- 08:50 AM Revision 2583: sql_gen.py: is_safe_name(): Fixed bug where keywords were incorrectly considered safe
- 08:40 AM Revision 2582: strings.py: repr_no_u(): Fixed bug where "u" prefix was removed even in reprs of non-strings
- 08:32 AM Revision 2581: db_xml.py: into_table_name(): Removed no longer necessary handling of simple functions, which is now done by sql.into_table_name(). Ensure that rank params in functions (not tables) are not treated specially as hierarchical.
- 08:21 AM Revision 2580: sql.py: put_table(): If into == None: For function calls, include the arguments in the into table name
- 08:17 AM Revision 2579: sql_gen.py: to_name_only_col(): Support non-Col Code inputs
- 07:42 AM Revision 2578: sql_gen.py CompareCond.to_str(), callers of combine_conds(): Removed unnecessary grouping () to make SQL clearer
- 07:31 AM Revision 2577: sql_gen.py: Added combine_conds() and use it in Join.to_str() and sql.py mk_select()
- 07:18 AM Revision 2576: sql_gen.py Join.to_str(), sql.py mk_select(): Combining conditions: Don't add newlines where not needed, so that output is less vertically spread out
- 07:10 AM Revision 2575: sql_gen.py: is_safe_name(): Fixed bug where names starting with a digit were incorrectly considered safe
- 07:06 AM Revision 2574: sql.py: put_table(): Separate temp table names from into table name with "_" instead of "-" so that quoting the table name will usually be unnecessary
- 07:03 AM Revision 2573: sql.py: esc_name_by_module(): Remove unused param ignore_case
- 06:59 AM Revision 2572: sql_gen.py: esc_name(): If is_safe_name(), just return name, to avoid escessive escaping in debug output for Redmine
- 06:55 AM Revision 2571: sql_gen.py: is_safe_name(): Don't consider uppercase letters safe because they would cause inconsistent behavior in PostgreSQL if quoted vs. not quoted (only unquoted identifiers are case-insensitive)
- 06:51 AM Revision 2570: sql.py: Removed no longer needed check_name()
- 06:50 AM Revision 2569: sql.py: esc_name_by_module(): psycopg2: If ignore_case is set but name is unsafe, just escape it instead of raising an exception
- 06:49 AM Revision 2568: sql_gen.py: Added is_safe_name()
- 06:39 AM Revision 2567: sql.py: put_table(): col_ustr(): Removed no longer needed sql_gen.as_Col() because mapping and join_cols now ensure that their contents are sql_gen.Col objects
06/01/2012
- 08:29 PM Revision 2566: schemas/functions.sql: Added _alt relational function
- 08:28 PM Revision 2565: sql.py: put_table(): Make mapping and join_cols a sql_gen.ColDict so that literal values will always be turned into sql_gen.Col objects. DuplicateKeyException: Use dict_subset_right_join() instead of dict_subset() so that all columns in a constraint are included in joins on out_table (such as for a relational function with omitted arguments).
- 08:25 PM Revision 2564: sql_gen.py: Added ColDict
- 08:19 PM Revision 2563: sql_gen.py: as_Col(): Added optional name param to specify that non-Col input will be renamed using NamedCol with the given name
- 07:06 PM Revision 2562: sql.py: put_table(): FunctionValueException: Fixed bug where only function calls, not plain columns, were handled, by using sql_gen.unwrap_func_call() to remove any function call *only if* there was one
- 07:04 PM Revision 2561: sql_gen.py: Added unwrap_func_call()
- 06:47 PM Revision 2560: bin/map: by_col: Stripping XML functions not in the DB: Fixed bug where preserve_funcs.add() was used when `preserve_funcs |=` should have been used to add the entire iterable that sql.tables() returns
- 06:45 PM Revision 2559: sql.py: not_null_col: Changed value to 'not_null_col' so that column doesn't seem like a status indicator of whether some value is not null (in fact it's just a column that is always not null)
- 06:05 PM Revision 2558: xml_func.py: Replaced xpath.get_1() with xpath.get_value() where possible, for simplicity
- 05:59 PM Revision 2557: xml_func.py: strip(): Evaluate structural functions like _ignore and _ref by process() instead of removing them. Store structural functions' names in structural_funcs module var. This ensures that _ref targets are still expanded in column-based import.
- 05:56 PM Revision 2556: xpath.py: get(): Create attrs: Put keys last so that any lookahead assertion's path will be created last as it would have without the assertion. This ensures that any value argument of an XML function will always go last even if a lookahead assertion would otherwise have caused it to be created with the element's keys, which previously were created before the attributes.
- 04:55 PM Revision 2555: sql.py: put_table(): If is_func, default into table name ends in () instead of '-pkeys'
- 04:54 PM Revision 2554: schemas/vegbien.sql, functions.sql: Made cast functions STRICT to enable the RETURNS NULL ON NULL INPUT optimization
- 04:33 PM Revision 2553: db_xml.py: put_table(): Pass is_func to sql.put_table()
- 04:32 PM Revision 2552: sql.py: put_table(): Added is_func param for whether out_table is the name of a SQL function, not a table
- 04:09 PM Revision 2551: db_xml.py: put_table(): Treat every node name that starts with "_" as a function, not just members of put_table_special_funcs. This ensures that DB function args are always treated as values, not children with fkeys to parent.
- 03:40 PM Revision 2550: bin/map: by_col: Strip only XML functions that are not in the DB
- 03:39 PM Revision 2549: db_xml.py: put_table(): Make special_funcs externally available as module constant put_table_special_funcs
- 03:38 PM Revision 2548: sql.py: tables(): Changed schema param to schema_like and filter the schema using LIKE so that all schemas can be selected
- 03:15 PM Task #343: integrate TNRS into VegBIEN
- This involves translating existing validation utilities to Python/Postgres
- 03:14 PM Task #427 (Resolved): Load all plots data
- 03:14 PM Task #426 (Resolved): Reload DB using column-based import
- 03:14 PM Task #425 (Resolved): Reimplement row-based logging mechanism for column-based import
- 03:13 PM Task #424 (New): Finish translating XML functions to SQL functions for column-based import
- * -_map-
* _range*
* -_avg-
* _compass
* a few others - 03:12 PM Task #386: load Canadensys data
- Half of the Canadensys datasources have been loaded
- 03:12 PM Task #325 (Resolved): map CTFS data
- See "*CTFS VegX map spreadsheet*":https://projects.nceas.ucsb.edu/nceas/projects/bien/repository/raw/inputs/CTFS/maps...
- 03:09 PM Task #366: refactor VegX
- See Format section of [[VegX]] schema
- 03:07 PM Task #348 (Resolved): 1st draft of schema
- 03:05 PM Task #371 (Resolved): formalize proposed changes to VegX
- 01:56 PM Revision 2547: to_do/timeline.doc: Updated to reflect the month we spent on optimization and column-based import
- 12:54 PM Revision 2546: sql.py: put_table(): in_table name: Remove '-pkeys' suffix from the into table name before adding '-input' so that the name is shorter and clearer
- 12:43 PM Revision 2545: sql.py: put_table(): Wrap repr() calls for debug messages in strings.as_tt() to add Redmine formatting
- 12:39 PM Revision 2544: sql.py: put_table(): Output "Adding index" debug message with level=2.5 so it's not part of the Redmine steps
05/31/2012
- 03:39 PM Revision 2543: schemas/vegbien.sql, functions.sql: Cast functions: Fixed bug where invalid value exceptions were not being caught, because implicit conversions to the return type apparently only happen outside the block containing the RETURN statement (i.e. at the end of the function). Fixed by adding explicit type conversion to return type, so that type conversion would happen inside try block.
- 03:31 PM Revision 2542: sql.py: put_table(): Re-enabled FunctionValueException handling, by just filtering out the value on all input columns that use the named function (since the error message does not specify which column it was that had the invalid value). This is in some ways better, anyway, because that way the invalid value is filtered out right away in all columns that could contain it, instead of potentially once for each column (if the value appears in more than one input column).
- 03:18 PM Revision 2541: sql.py: add_index(): Fixed bug where expressions could not be converted to a string until their table name had been removed
- 03:17 PM Revision 2540: sql_gen.py: Added Expr
- 03:13 PM Revision 2539: sql.py: add_index(): Fixed bug where expressions needed to be enclosed in () to distinguish them from plain columns
- 03:06 PM Revision 2538: sql.py: add_index(): Support simple expressions as well as columns
- 02:37 PM Revision 2537: sql.py: Renamed index_col() to add_index() so its name isn't similar to index_cols()
- 02:33 PM Revision 2536: sql_gen.py: FunctionCall: Removed __repr__() because it's a Code object and its to_str() does not take extra arguments
- 02:12 PM Revision 2535: sql.py: run_query(): FunctionValueException: Expanded parsing to include regular function calls, not just relational functions' trigger functions. put_table(): Disabled FunctionValueException handling because this expands FunctionValueException beyond what put_table() could handle.
- 01:38 PM Revision 2534: sql.py: put_table(): MissingCastException: Fixed bug where renaming of cast literal value was not properly propagated to the returned value of the function call, causing the query to assume that a DISTINCT ON column referred to column in one of the joined tables instead of a named column in the SELECT columns list. This logic error would have been very difficult to catch without inspecting the code!
- 01:33 PM Revision 2533: sql_gen.py: Added wrap_in_func()
- 01:25 PM Revision 2532: sql_gen.py: FunctionCall: Filter args through remove_col_rename() to remove any renamings from the function args
- 01:20 PM Revision 2531: sql.py: put_table(): No handler for exception: Print full exception instead of just first line to assist in debugging
- 01:06 PM Revision 2530: schemas/vegbien.sql, functions.sql: Removed _to* relational functions because type casting for those types is now automatic
- 01:02 PM Revision 2529: mappings/DwC2-VegBIEN.specimens.csv: Removed _to* relational functions because type casting for those types is now automatic
- 12:59 PM Revision 2528: schemas/functions.sql: Added cast functions for _to* relational functions
- 12:58 PM Revision 2527: schemas/vegbien.sql: Changed cast functions' input types to text because type must match exactly, not just be implicitly castable
- 12:47 PM Revision 2526: sql.py: run_query(): MissingCastException parsing: Support multiple-word types
- 12:38 PM Revision 2525: sql.py: put_table(): Handle MissingCastExceptions by attempting to call a function with the name of the type on the column
- 12:33 PM Revision 2524: sql_gen.py: Added Functions section with Function and FunctionCall
- 11:56 AM Revision 2523: sql.py: Added MissingCastException and parse it in run_query()
- 11:36 AM Revision 2522: schemas/vegbien.sql: Added cast functions for enum types which map invalid values to NULL
- 10:57 AM Revision 2521: sql.py: put_table(): Fixed bug where some exceptions with no handler would not even allow insertion of no rows into the out_table (due to type mismatch issues), by creating an empty pkeys table as a special case
- 10:49 AM Revision 2520: sql.py: put_table(): Preparing to insert new rows: Fixed bug where main_select needed to be generated *after* distinct_on was set in the if statement
- 10:48 AM Revision 2519: sql.py: put_table(): log_exc(): Fixed bug where the exception strings rather than the exceptions themselves needed to be put in the set, because exceptions are not comparable with ==
- 10:25 AM Revision 2518: sql.py: put_table(): Moved mk_main_select() call out of try block since it is not related to the exceptions that may be thrown
- 10:17 AM Revision 2517: sql.py: put_table(): log_exc(): Check if exception already caught before to avoid infinite loops
- 09:35 AM Revision 2516: Added debug2redmine and helper file debug2redmine.csv
- 09:20 AM Revision 2515: sql.py, db_xml.py: Removed unnecessary calls to sql_gen.clean_name() now that str() handles this automatically
- 09:14 AM Revision 2514: sql_gen.py: sql_gen classes inherit from new base class BasicObject, whose __str__() calls clean_name() on the object's repr(). Changed the main debug-repr producing method to be repr() instead of str().
- 08:45 AM Revision 2513: Moved clean_name() from sql.py to sql_gen.py because it's DB-general and so that it can be used by sql_gen.py without circular dependencies
- 08:41 AM Revision 2512: db_xml.py: into_table_name(): Handle hierarchical tables specially by including their rank in the into table. Interpret any table with a value column as a function, regardless of out_table name.
05/30/2012
- 11:07 PM Revision 2511: sql.py: put_table(): Log "Default value column does not exist in mapping" error with level 2.1 so that it doesn't appear in Redmine output
- 11:05 PM Revision 2510: db_xml.py: put_table(): Pass next as sql.put_table()'s default param now that it is supported
- 11:04 PM Revision 2509: sql.py: put_table(): Changed default param to be an *output* column because that is what would be passed in by db_xml.put_table(), and because there is already a mapping that resolves that to a flattened input column
- 10:37 PM Revision 2508: sql.py: put_table(): Added default param for the value or input column to use as the pkey for missing rows
- 10:20 PM Revision 2507: sql.py: put_table(): Use single quotes rather than double quotes around strings where possible
- 10:18 PM Revision 2506: db_xml.py: Added internal next param used by _simplifyPath. put_table_(): Refactored to use outer parent_ids_loc var and modify that as needed rather than having to pass parent_ids_loc as a param to put_table_().
- 09:55 PM Revision 2505: sql.py: put_table(): When calling strings.as_*table(), pass custom ustr that removes col renames and adds double quotes on plain strings
- 09:53 PM Revision 2504: strings.py: as_*table(): Added ustr param to override the method (by default ustr()) used to convert each value to a string
- 09:15 PM Revision 2503: sql_gen.py: MockDb.esc_value(): Use new strings.repr_no_u()
- 09:14 PM Revision 2502: strings.py: Added repr_no_u()
- 09:09 PM Revision 2501: sql.py: clean_name(): Also remove '`' (which is used by MySQL)
- 09:06 PM Revision 2500: sql.py: esc_name_by_module(): Use new sql_gen.esc_name()
- 09:03 PM Revision 2499: sql_gen.py: Added esc_name() and use it in MockDb.esc_name()
- 09:00 PM Revision 2498: sql.py: next_version(): Use special chars in version part of name string for clarity
- 08:53 PM Revision 2497: sql.py: mk_insert_select(): embeddable: function_name is first line of query for clarity, and to reduce length from including the column names. This also fixes the problem of double quotes around column names in the previous function_name.
- 08:47 PM Revision 2496: sql.py: esc_name_by_module(): Double embedded quotes to escape them instead of removing them
- 08:35 PM Revision 2495: sql.py: put_table(): Use "-" to separate temp table suffixes from into table name
- 08:26 PM Revision 2494: db_xml.py: into_table_name(): Format relational functions' into table names as a function call on the value column, using special chars for readability
- 08:19 PM Revision 2493: sql.py: run_query(): Exception parsing: Use "(.+?)" wherever possible to match names containing special chars
- 07:52 PM Revision 2492: sql.py: clean_name(): For clarity, just remove '"'s, so that "."s are preserved and show the path structure of the input name
- 07:38 PM Revision 2491: db_xml.py: put_table(): sql.put_table(): Name the into table ...literal instead of ...value if the value column is a literal value
- 07:08 PM Revision 2490: bin/map: Logging: log(): Remove extra debug info from DB query messages and format level 1.5 (summary) messages as Redmine list items
- 06:50 PM Revision 2489: sql.py: put_table(): Renamed temp_prefix param to into and allow it to be a sql_gen.Table object. Use into directly as the pkeys table, and make its default value be `out_table.name+'_pkeys'`.
- 06:31 PM Revision 2488: db_xml.py: put_table(): Pass custom temp_prefix to sql.put_table() for relational funcs, so that their value param's input column name is included in the temp table name
- 06:19 PM Revision 2487: sql.py: put_table(): Added optional param temp_prefix for the prefix of generated temp tables
- 06:13 PM Revision 2486: sql.py: put_table(): Made debug messages more self-documenting
- 05:44 PM Revision 2485: sql.py: put_table(): Changed "Setting missing rows' pkeys to NULL" to "Setting pkeys of missing rows to NULL" to avoid having single quote in debug output, which messes up text editor SQL syntax highlighting
- 05:40 PM Revision 2484: sql.py: Parsed exceptions: Use strings.as_tt() to format Python values
- 05:37 PM Revision 2483: strings.py: Split as_table() into as_table() and as_inline_table() depending on whether the table needs to be inlined in an ordered list item or not
- 05:36 PM Revision 2482: strings.py: Split as_table() into as_table() and as_inline_table() depending on whether the table needs to be inlined in an ordered list item or not
- 05:03 PM Revision 2481: strings.py: as_table(): Changed to use <pre> formatting because Redmine tables can't be embedded in ordered lists without restarting the numbering
- 03:58 PM Revision 2480: strings.py: as_table(): Fixed bug where table was not ended properly, by adding a space after the last \n and having rstrip() string only newlines
05/29/2012
- 09:19 PM Revision 2479: sql.py: mk_select(): Columns: Separate columns with newlines
- 09:10 PM Revision 2478: sql.py: put_table(): Use new strings.as_table() to format mappings as tables
- 09:09 PM Revision 2477: strings.py: Added as_tt() and as_table()
- 09:09 PM Revision 2476: bin/map: Logging: log(): Strip trailing newlines from msg
- 08:40 PM Revision 2475: strings.py: as_code(): Added multiline param to disable multiline formatted output
- 08:33 PM Revision 2474: sql.py: put_table(): "Ignoring existing rows, comparing on" debug message: Wrap the mapping in strings.as_code() so it will have Redmine syntax-highlighting
- 08:26 PM Revision 2473: sql.py: put_table(): "Putting columns" debug message: Wrap the mapping in strings.as_code() so it will have Redmine syntax-highlighting
- 08:22 PM Revision 2472: sql.py: DbConn.run_query(): Query debug message: Wrap the query in strings.as_code() so it will have Redmine syntax-highlighting
- 08:20 PM Revision 2471: strings.py: Added as_code()
- 08:04 PM Revision 2470: sql.py: DbConn.run_query(): Prepend "DB query" before the query debug message so it can be identified as a DB query
- 07:43 PM Revision 2469: db_xml.py: put_table(): Subset in_table: Document that in_table will be shadowed (hidden) by the created temp table, rather than versioned, now that the table is (almost) always created as a temp table
- 07:40 PM Revision 2468: sql.py: Create temp items as permanent in autocommit mode rather than in debug mode so that temp items are only permanent if actually committing result. This ensures that the generated SQL in test mode matches what would actually get run in regular commit mode, and the SQL is only altered to make the temp items visible if actually debugging (autocommit mode).
- 07:30 PM Revision 2467: sql.py, sql_gen.py: Reformatted generated SQL for presentability by adding newlines
- 07:14 PM Revision 2466: sql.py: DbConn.run_query(): Put a newline before the query in the debug message so that multiline queries have all rows at the left edge rather than the first row prefixed by other text
- 07:09 PM Revision 2465: sql.py: DbConn.run_query(): Don't put generated query debug message all on one line, so that embedded newlines are preserved
- 06:59 PM Revision 2464: sql.py: Fixed bug where queries with versioned identifiers which threw an exception (not related to name collisions) were being output with a too-high log_level, because *all* exceptions were output with the higher exc_log_level, by making the following changes: DbConn.run_query(): Changed exc_log_level param to log_ignore_excs param so that only certain exceptions would cause the query to be output with a higher log_level. Moved the code that actual emits the query debug message from DbConn.run_query() to module-level run_query() so it would apply the log_ignore_excs filter after the exception had already been parsed into specific types.
- 03:16 PM Revision 2463: Moved "Putting columns" debug message from db_xml.py put_table() to sql.py put_table() to put it in the same place as the other debug messages
- 03:12 PM Revision 2462: sql_gen.py: Added remove_col_rename() and use it where `if isinstance(value, NamedCol): value = value.code` was used
- 03:10 PM Revision 2461: sql_gen.py: CompareCond.to_str(): If left_value has been renamed as a NamedCol, unwrap it
- 02:53 PM Revision 2460: sql_gen.py: Join.to_str(): Fixed bug where USING should be used if all columns are join_same_not_null, rather than join_same, because USING uses plain = for comparison. sql.py: put_table(): input_joins now can use sql_gen.join_same_not_null in order to use USING syntax.
05/25/2012
- 07:14 PM Revision 2459: db_xml.py: put_table(): Output debug messages with a level of 1.5 to match sql.put_table()'s level for summary messages
- 07:01 PM Revision 2458: bin/map: Fixed bug where verbosity needed to be 1 outside of test mode so that profiling and errors stats would be printed at end of import. Verbosity defaults to 0.5 rather than 1 in test mode so profiling and errors stats do not clutter up the test output when running automated tests.
- 06:55 PM Revision 2457: bin/map: Only display verbose_errors in test mode, but with any nonzero verbosity. They should not be displayed outside of test mode because verbose errors make the log files huge.
- 06:52 PM Revision 2456: bin/map: Renamed verbose param to verbosity because it's now a number, not a boolean
- 06:51 PM Revision 2455: bin/map: Removed no longer used debug param (verbose=2 is used instead)
- 06:48 PM Revision 2454: bin/map: Fixed bug where verbose_errors' default value depended on debug var, which was not yet set. Removed verbose_errors param and instead turn verbose_errors on whenever verbosity >= 1. Verbosity defaults to 1 in test mode.
- 06:33 PM Revision 2453: bin/map: Logging: Don't set sql.run_raw_query.debug, because it is not used anymore (sql.connect(log_debug=...) is used instead)
- 06:29 PM Revision 2452: bin/map: Logging: Print debug messages (level > 1) prefixed with their level, to distinguish higher- and lower-level debug messages
- 06:22 PM Revision 2451: sql.py: put_table(): Only display warning for exceptions with no handler (which are unexpected), not missing mappings for NOT NULL columns (which are normal in datasources without those columns)
- 06:15 PM Revision 2450: sql.py: put_table(): Log summarizing debug messages with a level of 1.5 so they will be displayed even when the major SQL queries (which have a level of 2) are not shown
- 06:08 PM Revision 2449: bin/map: Provide a log_debug() function to sql.connect() if verbosity > 1 rather than >= 2, to support fractional verbosities
- 06:04 PM Revision 2448: sql.py: log_debug_none: Fixed bug where needed to take kw arg level to work with verbosity-based logging
- 05:57 PM Revision 2447: bin/map: Allow fractional verbosity values
- 05:56 PM Revision 2446: sql.py: Functions that version created tables, functions, etc. if they already exist: Use (default) exc_log_level=4 to hide the unsuccessful attempts to create items that already exist and show only the successful attempt
- 05:43 PM Revision 2445: sql.py: DbConn.run_query(): Added exc_log_level param to specify a different log_level if the query throws an exception. This will useful for functions that version created tables, functions, etc. if they already exist.
- 05:34 PM Revision 2444: sql.py: DbConn.run_query(): Removed no longer accurate doc comment, because that functionality is now in module-level run_query()
- 05:31 PM Revision 2443: sql.py: Specify log_levels for minor queries so they can be excluded from the debug output
- 05:16 PM Revision 2442: sql.py: select(): Pass log_level to run_query()
- 05:13 PM Revision 2441: sql.py: DbConn.run_query(): Added log_level param and pass it to self.log_debug(). run_query(): Pass extra kw_args to DbConn.run_query() (via run_raw_query()) so that caller can specify log_level.
- 04:54 PM Revision 2440: sql.py: run_query_into(): Fixed bug where "temporary tables cannot specify a schema name"
- 04:42 PM Revision 2439: bin/map: Switched to verbosity-level-based system of logging. verbose is now an integer, and debug sets the minimum verbosity to 2.
- 04:37 PM Revision 2438: input.Makefile: Configuration: Removed debug var since it's not used in the Makefile
- 04:09 PM Revision 2437: db_xml.py: put_table(): put_table_(): Fixed bug where row_ins_ct_ref needed to be passed recursively to put_table() as keyword arg, because the in_row_ct_ref is not passed recursively
- 04:07 PM Revision 2436: db_xml.py: put_table(): _simplifyPath: Parse "next" XPath param to extract col name of next level's pkey
- 03:26 PM Revision 2435: bin/map: by_col: xml_func.strip(): Don't remove _simplifyPath because it is now handled by db_xml.put_table()
- 03:25 PM Revision 2434: db_xml.py: put_table(): Added basic special handling for structural XML functions, which for now just skips the function
- 03:21 PM Revision 2433: xml_func.py: strip(): Added preserve param for XML functions not to remove
- 02:49 PM Revision 2432: db_xml.py: put_table(): Handle forward pointers in translation-to-sql_gen step instead of in XML-tree-parsing step, so that special handling for structural XML functions can use the parsed tree before any sql.put_table() processing takes place
- 02:44 PM Revision 2431: xml_dom.py: Added is_node()
- 02:22 PM Revision 2430: sql.py: table_row_count(): Pass start=0 to mk_select() to avoid "SELECT statement missing a WHERE, LIMIT, or OFFSET clause" warnings
- 02:12 PM Revision 2429: sql.py: put_table(): Handle unknown exceptions by returning NULL for all rows. Refactored Missing mapping for NOT NULL column handling to use new helper function remove_all_rows().
- 01:54 PM Revision 2428: sql.py: put_table(): Assert that insert_out_pkeys and insert_in_pkeys have same row count. Assert that pkeys and in_table have same row count.
- 12:57 PM Revision 2427: db_xml.py: put_table(): Use new sql.table_row_count()
- 12:56 PM Revision 2426: sql.py: Added table_row_count()
- 12:52 PM Revision 2425: db_xml.py: put_table(): Use new sql_gen.row_count
- 12:47 PM Revision 2424: sql_gen.py: Added row_count
- 12:41 PM Revision 2423: db_xml.py: put_table(): Count # rows and update in_row_ct_ref once all columns have been processed. Don't pass in_row_ct_ref to recursive calls because it should only be increased once.
- 12:28 PM Revision 2422: db_xml.py: put_table(): Added in_row_ct_ref param to store the # of input rows processed. Renamed row_ct_ref param to row_ins_ct_ref to distinguish it from new in_row_ct_ref param.
05/24/2012
- 09:26 PM Revision 2421: sql_gen.py: MockDb.esc_name(): Don't use sql.esc_name_by_module() to avoid circular dependency on sql module
- 09:20 PM Revision 2420: sql.py: put_table(): Factored out mk_select() calls in calls to run_query_into_pkeys() into new helper function insert_into_pkeys()
- 09:09 PM Revision 2419: sql.py: put_table(): run_query_into_pkeys() calls use order_by=None in their select statements because there is a pkey, so order (row #) does not matter
- 09:05 PM Revision 2418: db_xml.py: put_table(): Subset in_table if limit != None or start != 0. start param defaults to 0 again to avoid subsetting the table when starting from row 0 (with no limit).
- 08:46 PM Revision 2417: db_xml.py: put_table(): Don't pass limit, start recursively, because the table subsetting will happen only once in the first invocation of the function. Moved limit, start params to end since they are not passed recursively. start param no longer defaults to 0 because this is not needed since sql.put_table() now sets start to 0 where needed.
- 08:38 PM Revision 2416: sql.py: put_table(): Removed limit and start params because they were never fully implemented, and because it's simpler to just have the caller subset their input table
- 08:27 PM Revision 2415: lists.py: Added uniqify()
- 08:08 PM Revision 2414: sql.py: Moved mk_flatten_mapping(), flatten() to Basic queries section since they don't involve database structure info
- 08:06 PM Revision 2413: sql.py: put_table(): Use single quotes rather than double quotes around strings where possible
- 07:59 PM Revision 2412: schemas/functions.sql, vegbien.sql: Changed CAST-related relational functions to return NULL on data exceptions and convert the exceptions to warnings. This helps column-based import by mapping invalid values to NULL instead of aborting the whole query on the first invalid value.
- 07:33 PM Revision 2411: sql.py: index_col(): Cache the query so it doesn't try to add an index on the same column multiple times
- 07:18 PM Revision 2410: sql.py mk_select(), sql_gen.py Join.to_str(): Fixed bug where conditions needed to be wrapped in () before being AND-ed together to ensure the proper operator precedence
- 06:49 PM Revision 2409: sql.py: put_table(): Add index on columns with invalid values to enable fast filtering
- 06:47 PM Revision 2408: sql.py: Added index_col()
- 06:18 PM Revision 2407: sql.py: put_table(): Add pkey on returned pkeys table to enable fast joins
- 06:17 PM Revision 2406: sql.py: Added index_pkey()
- 05:41 PM Revision 2405: sql.py: mk_update(): When running sql_gen.to_name_only_col(), check that the col's table is table
- 05:38 PM Revision 2404: sql.py: put_table(): Renamed *_pkeys to insert_*_pkeys to distinguish them from the full set of pkeys on the input table
- 05:27 PM Revision 2403: sql.py: put_table(): FunctionValueException: Change invalid values to NULL using UPDATE instead of filtering them out using WHERE, to avoid adding lots of conditions to the SELECT statement
- 05:11 PM Revision 2402: sql.py: Added mk_update() and update()
- 05:10 PM Revision 2401: sql_gen.py: Added to_name_only_col()
- 04:56 PM Revision 2400: sql_gen.py: Added as_Value()
- 04:29 PM Revision 2399: sql.py: mk_select(): conds: Use new sql_gen.ColValueCond instead of sql_gen.as_ValueCond(). Documented that Code and ValueCond are sql_gen objects.
- 04:28 PM Revision 2398: sql_gen.py: Added ColValueCond
- 03:59 PM Revision 2397: sql.py: mk_flatten_mapping(): Filter str(col) through clean_name() to remove quotes, etc.
- 03:58 PM Revision 2396: sql.py: Added clean_name()
- 03:43 PM Revision 2395: sql.py: put_table(): Join together input tables into new table for speed and so don't modify input if values edited
- 03:37 PM Revision 2394: sql.py: mk_flatten_mapping(): Take as_items param to return a list of dict items instead of a dict. Sort preserve cols before other cols. flatten(): Turn on as_items so that cols list is sorted in input order, with preserve cols first. This ensures that if a pkey is provided in preserve, it will be the first col in the generated table.
- 03:24 PM Revision 2393: sql.py: mk_flatten_mapping(), flatten(): Take list of cols to select instead of using all cols in all tables to join
- 02:58 PM Revision 2392: sql.py: mk_flatten_mapping(), flatten(): Renamed flat_table param to into to be consistent with run_query_into() and put it first because it is the output param
- 02:55 PM Revision 2391: sql.py: Added flatten()
- 02:38 PM Revision 2390: sql.py: mk_flatten_mapping(): preserve Col objects will have tables changed to flat_table to work with flattened table
- 02:29 PM Revision 2389: sql.py: mk_flatten_mapping(): Added preserve param for list of columns not to rename
- 02:18 PM Revision 2388: sql.py: esc_name_by_module(): Support module value None, and use default module psycopg2 for it
05/23/2012
- 09:58 PM Revision 2387: sql.py: put_table(): Renamed *pkeys_ref to *pkeys to reflect that they are now objects rather than an array-based references
- 09:54 PM Revision 2386: sql.py: run_query_into(): Renamed into_ref param to into to reflect that it's now an object rather than an array-based reference
- 09:51 PM Revision 2385: sql.py: run_query_into(): Made into_ref a sql_gen.Table instead of an array containing a table name to improve flexibility and clarity
- 09:34 PM Revision 2384: dicts.py: Added join()
- 09:20 PM Revision 2383: sql.py: Added mk_flatten_mapping()
- 08:28 PM Revision 2382: sql.py: put_table(): Renamed the copy of in_tables that gets modified to in_tables_, so that the original list can eventually be reused in joining together the input tables into a temp table
- 07:10 PM Revision 2381: sql.py: run_query(): FunctionValueException: Also match "date/time field value out of range" errors
- 07:04 PM Revision 2380: sql.py: put_table(): conds: Use a set instead of a list for faster checking of the "cond not in conds" assertion
- 06:55 PM Revision 2379: sql.py: mk_select(): conds: Support containers of any iterable type
- 06:52 PM Revision 2378: sql.py: put_table(): Made conds a list so that there can be multiple conditions on the same column
- 06:36 PM Revision 2377: sql.py: mk_select(): conds is list of (key, value) tuples instead of dict (dict still supported for compatibility), so that there can be multiple conditions on the same column
- 06:35 PM Revision 2376: sql.py: mk_select(): conds is list of (key, value) tuples instead of dict (dict still supported for compatibility), so that there can be multiple conditions on the same column
- 06:28 PM Revision 2375: util.py: NamedTuple inherits from objects.BasicObject so that it's comparable and hashable. This fixes a bug in dicts.make_hashable() where the NamedTuple created for a dict would appear to be hashable but would always compare as unequal.
- 06:15 PM Revision 2374: sql.py: DbConn.esc_value(): Run strings.to_unicode() on the generated string so that if it contains unescaped non-ASCII characters, these will not cause problems when concatenated with plain strings
- 05:58 PM Revision 2373: sql.py: run_query(): FunctionValueException: Unpack match.groups() into vars to make code clearer
- 05:56 PM Revision 2372: exc.py: str_(): Avoid traceback exception-formatting functions when possible because they escape non-ASCII characters
- 05:11 PM Revision 2371: sql.py: get_cur_query(): If no raw query: Use strings.ustr() instead of repr() to ensure that if the exception is parsed, embedded quotes will not be double-escaped. Prefix the query by [input] to show that it's not the raw query.
- 04:59 PM Revision 2370: sql_gen.py: Non-Code objects: __str__() passes informative placeholder string to self.to_str() instead of empty string
- 04:41 PM Revision 2369: sql.py: ExceptionWithNameValue: Use repr() instead of strings.ustr() on the value
- 04:38 PM Revision 2368: sql.py: run_query(): Exception parsing: Use non-greedy qualifier "?" in regexps wherever possible to avoid matching closing quotes later in the error message
- 04:32 PM Revision 2367: sql_gen.py: MockDb.esc_value(): Use repr() instead of strings.ustr() so the quotes around the value are included
- 04:30 PM Revision 2366: sql_gen.py: ValueCond and Join class hierarchies inherit from objects.BasicObject like Code does
- 04:24 PM Revision 2365: sql.py: put_table(): ignore(): Fixed bug where value needed to be filtered through repr(). NullValueException: Fixed bug where value passed to ignore() was the string 'NULL' instead of the value None.
- 04:14 PM Revision 2364: mappings/DwC2-VegBIEN.specimens.csv: plantname.rank: Filter through _toTaxonrank
- 04:03 PM Revision 2363: sql.py: put_table(): ignore(): Avoid infinite loops by asserting that in_col is not in conds
- 03:58 PM Revision 2362: objects.py: BasicObject: Fixed bug where util needed to be imported. Added __eq__() and __hash__().
- 03:47 PM Revision 2361: strings.py: Removed no longer used DebugPrintable (that functionality is now in objects.BasicObject)
- 03:46 PM Revision 2360: sql_gen.py: Code: Inherit from new objects.BasicObject
- 03:46 PM Revision 2359: Added objects.py
- 03:37 PM Revision 2358: sql.py: put_table(): Renamed log_ignore() to ignore() and factored common conds-modifying code into it
- 03:29 PM Revision 2357: sql.py: put_table(): Moved post-insert code outside while loop because it will now always be run (there are no longer special cases where the postprocessing doesn't happen)
- 03:25 PM Revision 2356: sql.py: put_table(): Missing mapping for NOT NULL column: Just create an empty pkeys table, since the missing rows' pkeys will be set to NULL later
- 03:17 PM Revision 2355: sql.py: put_table(): Joining together output and input pkeys: Use new sql_gen.join_same_not_null
- 03:14 PM Revision 2354: sql.py: put_table(): Setting missing rows' pkeys to NULL: Use new sql_gen.join_same_not_null
- 03:14 PM Revision 2353: sql_gen.py: Join: Added join_same_not_null. to_str(): Refactored to switch order of left and right tables and cols because left_table is on the right in the comparison, and using the sides of the comparison instead of the sides of the join makes the code clearer.
- 02:51 PM Revision 2352: sql_gen.py: Renamed join_using to join_same to reflect that it can also be used without USING
- 02:48 PM Revision 2351: sql.py: put_table(): Set missing rows' pkeys to NULL
- 02:10 PM Revision 2350: sql.py: put_table(): NullValueException: no mapping for missing col: Fixed bug where run_query_into_pkeys() was still using insert_joins instead of input_joins
- 02:06 PM Revision 2349: sql_gen.py: Added MockDb. All __str__() methods: Use self.to_str() with mockDb.
- 01:59 PM Revision 2348: sql_gen.py: Use db.esc_name() instead of sql.esc_name(db, ...) so passed-in db can be a mock object
- 01:58 PM Revision 2347: sql.py: DbConn: Added esc_name()
- 01:51 PM Revision 2346: db_xml.py: put_table(): Debug-print which columns are being put
- 01:50 PM Revision 2345: sql.py: ConstraintException, NullValueException: Improved error messages
- 01:31 PM Revision 2344: sql.py: put_table(): FunctionValueException: Fixed bug where out_table was still assumed to be an escaped string, but is now a Table object
- 01:29 PM Revision 2343: sql.py: mk_select(): joins: Use new table_not_null_col() instead of pkey() to get a non-NULL column to filter out on
05/22/2012
- 10:00 PM Revision 2342: exc.py: add_msg(): Fixed bug where msg needed to be converted to a unicode object before appending it to another unicode object
- 09:54 PM Revision 2341: mappings/VegX-VegBIEN.stems.csv: Fixed bug where taxonfit was named taxonFit. (This was only recently discovered because column names are now escaped, causing them not to be case-insensitive.)
- 09:51 PM Revision 2340: sql.py: Added table_not_null_col()
- 09:50 PM Revision 2339: sql.py: Added table_cols() and use it in pkey()
- 09:36 PM Revision 2338: schemas/vegbien.sql, schemas/functions.sql: Relational functions: Added dummy not_null column to provide a column to use in LEFT JOIN filter-out filters
- 09:24 PM Revision 2337: sql.py: mk_insert_select(): embeddable: Use new sql_gen.NamedTable
- 09:23 PM Revision 2336: sql_gen.py: Added NamedTable. Table: Added to_Table().
- 09:06 PM Revision 2335: sql_gen.py: Added section labels for each type of SQL code object
- 08:25 PM Revision 2334: sql.py: put_table(): DuplicateKeyException: Fixed bug where dict_subset_right_join() was used instead of dict_subset(), adding spurious None values for columns in the constraint which are not in the input tables
- 08:23 PM Revision 2333: sql_gen.py: as_Col(): Don't allow None cols
- 08:06 PM Revision 2332: schemas/vegbien.ERD.mwb: Synced with schemas/vegbien.sql
- 07:39 PM Revision 2331: sql.py: Removed no longer used clean_name()
- 07:38 PM Revision 2330: sql.py: mk_insert_select(): embeddable: Removed clean_name() because the function name is now escaped where it's used
- 07:36 PM Revision 2329: sql.py: put_table(): Added support for out_table values that are Table objects
- 07:31 PM Revision 2328: sql.py: mk_insert_select(): Fixed bug where table for creating the returning column Col object was the already-escaped string, instead of the Table object
- 07:24 PM Revision 2327: sql.py: mk_insert_select(): Fixed bug where function name and returning col were not being escaped
- 07:08 PM Revision 2326: sql.py: put_table(): log_ignore(): Fixed bug where in_col needed to be passed through str() because it's a column object
- 07:03 PM Revision 2325: sql.py: put_table(): Fixed bug where the filter_out join should only be used in the insert, not in the select of existing/inserted rows. insert_select() call: Fixed compatibility bug where old versions of Python did not support mixing keyword args and ** args.
- 06:32 PM Revision 2324: sql.py: put_table(): Fixed bug where "add_row_num(db, out_pkeys_ref[0])" was mistakenly put under the "if row_ct_ref != None" if statement
- 06:26 PM Revision 2323: sql_gen.py: Renamed NamedCode to NamedCol to better reflect its specific use
- 06:23 PM Revision 2322: sql.py: Removed unnecessary calls to check_name()
- 06:22 PM Revision 2321: sql.py: mk_insert_select(): Fixed bug where returning col was not being escaped
- 06:20 PM Revision 2320: sql.py: add_row_num(): Fixed bug where table name was not being escaped
- 06:13 PM Revision 2319: sql.py: run_query_into(): Fixed bug where into table name was not being escaped
- 06:07 PM Revision 2318: sql.py: mk_insert_select(): Fixed bug where utput column names were not being escaped
- 05:57 PM Revision 2317: sql.py: put_table(): Fixed bug where only string columns were being included in the distinct_on, but columns are now always sql_gen.Col instances
- 05:53 PM Revision 2316: sql.py: put_table(): Put together varying insert_select() args using dict instead of individual vars
- 05:51 PM Revision 2315: sql.py: mk_select(): Fixed bug where order_by needed to default to None if distinct_on was used. Fixed bug where cond values were being treated as %s params in addition to being parsed by sql_gen.as_ValueCond().to_str().
- 05:40 PM Revision 2314: sql_gen.py: Col: Added to_Col()
- 05:31 PM Revision 2313: db_xml.py: put_table(): Accept sql_gen.Table objects or strings instead of separate table and schema names
- 05:10 PM Revision 2312: sql.py: put_table(): Require all in_table_cols to be sql_gen.Col objects
- 05:03 PM Revision 2311: sql_gen.py: ValueCond: Unwrap NamedCode objects
- 04:55 PM Revision 2310: sql_gen.py: NamedCode: Inherit from Col so that its name can be retrieved using the same attribute as Col's
- 04:43 PM Revision 2309: sql.py: put_table(): Debug-log each caught exception
- 04:41 PM Revision 2308: exc.py: str_(): Added first_line_only param to return just the first line
- 04:26 PM Revision 2307: sql.py: ConstraintException: Changed text of message to specify that a constraint was violated
- 04:14 PM Revision 2306: sql.py: Renamed ExceptionWithColumns to ConstraintException and added name field to contain the constraint name, if any
- 04:06 PM Revision 2305: sql.py: put_table(): If there are join_cols, don't get output pkeys of inserted rows and instead select all rows (existing and inserted) after the insert
- 04:04 PM Revision 2304: sql_gen.py: Join.to_str(): Fixed bug where order of right_table_col and left_table_col was reversed when applying as_ValueCond() and as_Col()
- 03:33 PM Revision 2303: sql.py: put_table(): Moved things outside of the try clause which should not produce the exceptions
- 03:21 PM Revision 2302: sql_gen.py: Code: Extend new strings.DebugPrintable instead of implementing __str__(), __repr__() itself
- 03:20 PM Revision 2301: strings.py: Added DebugPrintable
- 03:17 PM Revision 2300: sql_gen.py: Code: __str__(): Added class name. Added __repr__().
- 03:16 PM Revision 2299: util.py: Added class_name()
- 02:55 PM Revision 2298: sql_gen.py: Join.to_str(): Fixed bug in USING syntax where columns were not escaped
- 02:48 PM Revision 2297: sql.py: put_table(): Order selects by in_tables0's pkey to avoid undefined orderings on multiple runs of the same query
- 02:42 PM Revision 2296: sql.py: mk_select(): Removed no longer used esc_name_()
- 02:41 PM Revision 2295: sql_gen.py: as_Table() Removed no longer used support for (schema, table) tuples
- 02:39 PM Revision 2294: sql_gen.py: Removed no longer used unescape_table() and table2sql_gen()
- 02:38 PM Revision 2293: sql.py: mk_select(): Removed no longer used table_is_esc
- 02:37 PM Revision 2292: sql.py: mk_insert_select(): Removed no longer used table_is_esc
- 02:34 PM Revision 2291: sql.py: pkey(): Removed no longer used table_is_esc
- 02:31 PM Revision 2290: sql.py: cleanup_table(): Switched from table_is_esc to sql_gen.as_Table.to_str()
- 02:19 PM Revision 2289: csv2db: Switched to using plain table names rather than table_is_esc
- 02:13 PM Revision 2288: bin/map: Switched to using sql_gen rather than table_is_esc
- 02:05 PM Revision 2287: sql_gen.py: Removed no longer needed col2sql_gen() and value2sql_gen()
- 02:04 PM Revision 2286: sql.py: Replaced sql_gen.value2sql_gen() with sql_gen.as_Col()
- 02:00 PM Revision 2285: sql.py: Replaced sql_gen.col2sql_gen() with sql_gen.as_Col()
- 01:57 PM Revision 2284: sql.py: mk_select(): Inline cond() and don't use sql_gen.as_Col because sql_gen.as_ValueCond.to_str() calls it
- 01:54 PM Revision 2283: sql_gen.py: Removed no longer needed cond2sql_gen()
- 01:53 PM Revision 2282: sql.py: mk_select(): cond(): Parse conditions using sql_gen-only functions
- 01:47 PM Revision 2281: sql_gen.py: Removed no longer needed join2sql_gen()
- 01:44 PM Revision 2280: sql.py: put_table(): Switched joins to sql_gen.Join objects. mk_select(): Only accept joins which are sql_gen.Join objects.
- 01:38 PM Revision 2279: sql.py: put_table(): Removed no longer used table_is_esc param
- 01:36 PM Revision 2278: sql.py: put_table(): Switched joins to sql_gen.Join objects
- 01:28 PM Revision 2277: sql.py: mk_select(): joins: Switched to using sql_gen.Join.to_str() to render joins to SQL
- 01:24 PM Revision 2276: sql_gen.py: Join.to_str(): Fixed bugs revealed in first test of function
05/21/2012
- 11:05 PM Revision 2275: db_xml.py: put_table(): Turn off table_is_esc when calling sql.put_table() and don't escape out_table
- 11:04 PM Revision 2274: sql.py: mk_insert_select(): Use sql_gen.table2sql_gen().to_str() to escape the table
- 10:57 PM Revision 2273: db_xml.py: put_table(): First in_tables table is sql_gen.Table object
- 10:49 PM Revision 2272: db_xml.py: put_table(): Converted row (mapping) values to sql_gen objects
- 10:45 PM Revision 2271: sql.py: mk_select(): Accept main tables (table0's) that are Table objects. This change requires plain SQL code to be wrapped in a CustomCode object if it should not be unescaped and converted to a Table object.
- 10:42 PM Revision 2270: sql_gen.py: as_Table(): Accept tables that are Code objects, not just Table objects
- 10:40 PM Revision 2269: sql_gen.py: CustomCode: Fixed bug where needed to inherit from Code
- 10:19 PM Revision 2268: sql.py: put_table(): Return a sql_gen.Col object instead of an old-style tuple
- 10:00 PM Revision 2267: sql.py: mk_select(): joins: Switched to using filter_out as an attribute of the Join object instead of a sentinel value for the first column. Filter by the right table's pkey being NULL instead of each joined column being NULL, because some joined columns may contain NULL values which would mess things up, but the pkey presumably is NOT NULL.
- 09:56 PM Revision 2266: sql_gen.py: Join.to_str(): Fixed bug where type_ None was being concatenated with the JOIN str
- 09:31 PM Revision 2265: sql_gen.py: Join.to_str(): Fixed bug where USING syntax could not be used for filter_out join type, because a separate right column is required for filtering
- 09:20 PM Revision 2264: sql_gen.py: Use new table2sql_gen() in col2sql_gen(), join2sql_gen()
- 09:18 PM Revision 2263: sql.py: mk_select(): joins: Convert all joins to sql_gen format using join2sql_gen()
- 09:17 PM Revision 2262: sql_gen.py: Added table2sql_gen()
- 08:44 PM Revision 2261: sql_gen.py: Added join2sql_gen()
- 08:33 PM Revision 2260: sql_gen.py: Added as_Col(). as_ValueCond(): Added support for assuming the value is a column rather than a literal value, using the default_table param. Added Join.
- 07:10 PM Revision 2259: sql_gen.py: Put parameterized SQL code objects in separate section
- 07:08 PM Revision 2258: sql.py: put_table(): DuplicateKeyException: Assert that join_cols has changed to avoid infinite loops
- 06:59 PM Revision 2257: sql.py: put_table(): Moved getting pkeys of already existing rows from DuplicateKeyException to try clause, so that it always runs if there are join_cols. DuplicateKeyException: Add new duplicate key cols to join_cols instead of replacing join_cols so that multiple unique constraints being violated causes the union of their columns to be used for join_cols.
- 06:23 PM Revision 2256: sql_gen.py: Added CustomCode
- 06:05 PM Revision 2255: sql.py: mk_select(): joins: Fixed bug where joins dict was being modified without first being copied, causing the input value to be modified
- 05:52 PM Revision 2254: Compare object()-based sentinel values using is. Where sentinel values must be compared using ==, use rand.rand_int() instead.
- 05:13 PM Revision 2253: sql.py: put_table(): Added debug messages for every action performed
- 04:45 PM Revision 2252: sql.py: put_table(): Moved assignment of in_pkeys_ref outside loop so it wouldn't need to be re-versioned every iteration
- 04:42 PM Revision 2251: sql.py: put_table(): Changed temp_suffix to temp_prefix so all temp tables for a given out_table would have the same prefix. (Existing name collisions due to truncated names are not a problem because version prefixes are automatically added.)
- 04:23 PM Revision 2250: mappings/DwC2-VegBIEN.specimens.csv: Filter dates through _toTimestamp
- 04:20 PM Revision 2249: schemas/functions.sql: Added _toTimestamp
- 04:15 PM Revision 2248: mappings/DwC2-VegBIEN.specimens.csv: Filter coordsaccuracy through _toDouble
- 04:12 PM Revision 2247: sql.py: FunctionValueException parsing: Support values containing non-word and non-ASCII characters
- 04:11 PM Revision 2246: exc.py: Support exception messages containing non-ASCII characters
05/18/2012
- 07:10 PM Revision 2245: sql.py: put_table(): Print debug messages about how exceptions are being handled
- 06:45 PM Revision 2244: sql.py: put_table(): After getting pkeys of already existing rows, insert new rows
- 06:42 PM Revision 2243: sql.py: put_table(): Handle FunctionValueExceptions by excluding rows with the invalid value in their "value" column
- 06:41 PM Revision 2242: sql.py: run_query(): Also parse "invalid input *syntax* at assignment" errors as FunctionValueExceptions
- 06:39 PM Revision 2241: sql_gen.py: Col: Convert string table names to Table objects
- 06:09 PM Revision 2240: sql.py: run_query(): Parse "invalid input value at assignment" errors' values as well
- 05:55 PM Revision 2239: sql.py: run_query(): Parse "invalid input value at assignment" errors as FunctionValueExceptions
- 05:27 PM Revision 2238: sql.py: mk_select(): joins: filter_out: Pass NULLs through. Use sql_gen.*2sql_gen() to add the left and right table names to the columns.
- 05:26 PM Revision 2237: sql_gen.py: cond2sql_gen(): Take assume_col param and pass it to value2sql_gen()
- 04:45 PM Revision 2236: sql.py: put_table(): Use table-qualified pkey col names whenever possible, to avoid ambiguous column references
- 04:12 PM Revision 2235: mappings/DwC2-VegBIEN.specimens.csv: placenames: Convert ranks using _toPlacerank to work with multi-inserts
- 04:11 PM Revision 2234: sql.py: DbConn._db(): Fixed bug where the isolation level was not set to "SERIALIZABLE" in a portable way
- 04:04 PM Revision 2233: sql.py: mk_select(): distinct_on is turned off when distinct_on == [] rather than when it's None
- 03:48 PM Revision 2232: schemas/vegbien.sql: Added _toPlacerank
- 03:43 PM Revision 2231: schemas/vegbien.sql: Added _toTaxonrank
- 03:35 PM Revision 2230: sql.py: put_table(): Handle NullValueExceptions by removing invalid rows
- 03:31 PM Revision 2229: sql_gen.py: Added NamedCode
- 03:30 PM Revision 2228: sql_gen.py: Added __str__() to base classes for debugging
- 02:46 PM Revision 2227: sql.py: mk_select() (and sql_gen.py): Fixed bugs where literal strings were treated as literal values when they should have been treated as column names. Take default_table param to determine default table to use if a column doesn't have an explicit table. put_table(): mk_main_select(): Pass in_tables0 as mk_select()'s default_table.
- 12:54 PM Revision 2226: sql.py: mk_select(): cond(): Run additional sql_gen translation functions cond2sql_gen() and col2sql_gen() on the left and right sides of the comparison
- 12:50 PM Revision 2225: sql_gen.py: ValueCond: Fixed bug where values which are Code objects were being converted to Literals. Added cond2sql_gen().
05/17/2012
- 08:01 PM Revision 2224: sql.py: mk_select(): join(): Use cond() now that it supports sql_gen format
- 07:50 PM Revision 2223: sql_gen.py: Added col2sql_gen() and use it in value2sql_gen()
- 07:25 PM Revision 2222: sql_gen.py: CompareCond: By default, compare NULL values literally. Support operator values to pass NULLs through.
- 07:23 PM Revision 2221: strings.py: remove_prefix(), remove_suffix(): Added removed_ref param
- 06:28 PM Revision 2220: sql.py: mk_select(): parse_col(): Use sql_gen.value2sql_gen().to_str()
- 06:22 PM Revision 2219: sql_gen.py: Added as_Table(), unescape_table(), value2sql_gen()
- 03:37 PM Revision 2218: sql.py: mk_select(): Documented conds param
- 03:32 PM Revision 2217: sql.py: mk_select(): cond(): Switched to using sql_gen so that custom conds would be supported
- 03:19 PM Revision 2216: sql_gen.py: ValueCond.to_str(): Made value_code a Code object instead of a string, and renamed it to left_value to reflect where it goes. Added as_ValueCond().
- 03:11 PM Revision 2215: sql.py: esc_value(): Fixed bug where db needed to be referenced through self
- 02:22 PM Revision 2214: sql_gen.py: ValueCond.to_str(): Added value_code param
- 02:16 PM Revision 2213: sql_gen.py: Literal, CompareCond: Implemented to_str(). ValueCond: Autoconvert literal values to Literals.
- 02:14 PM Revision 2212: sql.py: DbConn: Added esc_value()
- 01:52 PM Revision 2211: Moved SQL code generation classes from sql.py to new sql_gen.py. sql_gen.py: Added Code, Literal, ValueCond, and CompareCond. sql.py: Removed Query because we will use a different approach.
- 12:43 PM Revision 2210: sql.py: Added Query, Table, Col
- 11:28 AM Revision 2209: sql.py: get(): Fixed bug where limit=1 needs to be passed to select() as a keyword arg now that the distinct_on param comes before it
- 11:01 AM Revision 2208: sql.py: put_table(): mk_main_select(): Pass outer var conds to mk_select()
- 10:57 AM Revision 2207: sql.py: put_table(): mk_select_(): Fixed bug where it was sometimes being called without distinct_on, causing it to return a different # of rows. Renamed mk_select_() to mk_main_select() for clarity.
- 10:48 AM Revision 2206: sql.py: put_table(): Do inserts and selects in a loop so that it will keep retrying the operation with additional constraints until it succeeds
05/15/2012
- 03:56 PM Revision 2205: sql.py: put_table(): mk_select_(): Fixed bug where order_by needed to be None because otherwise it wouldn't match the distinct_on cols if they were specified
- 03:55 PM Revision 2204: sql.py: put_table(): insert_(): Fixed bug where distinct_on was not passed to mk_select_()
- 03:30 PM Revision 2203: sql.py: put_table(): mk_select_(): Fixed bug where distinct_on needed to be passed as a keyword param
- 03:21 PM Revision 2202: sql.py: put_table(): insert_() and mk_select_() take distinct_on param
- 03:10 PM Revision 2201: sql.py: put_table(): Factored out code that inserts into pkeys table into run_query_into_pkeys() helper function
- 02:55 PM Revision 2200: sql.py: mk_select(): Implemented DISTINCT ON according to the distinct_on param
- 02:48 PM Revision 2199: sql.py: mk_select(): Added distinct_on param to set the columns to SELECT DISTINCT ON
- 02:31 PM Revision 2198: sql.py: clean_name(): Convert names to lowercase so that PostgreSQL will behave the same whether the name is escaped with "" or not. This will help avoid bugs in code that uses temp tables created by the sql module.
- 02:29 PM Revision 2197: sql.py: put_table(): Added order_by=None wherever rows were not supposed to be re-ordered. On DuplicateKeyException: Save existing pkeys in temp table for joining on.
- 01:31 PM Revision 2196: db_xml.py: put_table(): Pass limit and start to sql.put_table()
- 01:09 PM Revision 2195: db_xml.py: put_table(): Added limit and start options
- 11:54 AM Revision 2194: sql.py: When creating a temporary entity (table, function, etc.), instead create it as a permanent entity in debug mode so it can be viewed after the program is run
- 11:40 AM Revision 2193: sql.py: DbConn: Store whether in debug mode (log_debug != log_debug_none) for easy use by methods
- 11:31 AM Revision 2192: bin/map: connect_db(): Turn on autocommit mode in debug mode if commit is on, so that incremental results can be seen in the DB
- 11:30 AM Revision 2191: sql.py: DbConn: Use internal autocommit handling instead of DB connection autocommit attr to avoid autocommits inside a savepoint
- 11:15 AM Revision 2190: sql.py: DbConn: Added autocommit option to turn on autocommit mode. Use set_session() instead of SQL command to set isolation level.
05/14/2012
- 05:50 PM Revision 2189: sql.py: mk_insert_select(): embeddable: Fixed bug where the function may do different things when run, because the function (and other statements whose cached strings depend on the function name) may be run after the function definition would have changed, by versioning the function name and using CREATE FUNCTION instead of CREATE OR REPLACE FUNCTION so that its definition never changes
- 05:28 PM Revision 2188: sql.py: Parse "function already exists" errors as DuplicateFunctionException
- 05:13 PM Revision 2187: sql.py: mk_select(): joins: Fixed bug where join_not_equal did not do what it was designed for, which is filtering out matches of the join condition (before the bug fix, it effectively did a cross join with matching rows excluded, causing duplication of rows). Renamed join_not_equal to filter_out to reflect its intended use. Support table-scoped column names in the WHERE conds list.
- 04:22 PM Revision 2186: sql.py: put_table(): Fixed bug where ORDER BY column needed to have table0 name prefixed (if it didn't already have a table name), to avoid ambiguous column references
- 04:11 PM Revision 2185: sql.py: mk_select(): Fixed bug in joins where right_col had the table name prepended *before* it was copied for use with a different table name in join_using and join_not_equal
- 03:42 PM Revision 2184: Mapped some unmapped fields in DwC inputs
- 02:19 PM Revision 2183: Added mappings/for_review/DwC2-VegBIEN.specimens.fields.csv
- 01:21 PM Revision 2182: db_xml.py: put_table(): Fixed bug where didn't commit right after inserting node, but instead waited until children with fkeys to parent (independent of the node itself) were inserted
- 01:16 PM Revision 2181: sql.py: put_table(): insert_(): Use insert_select() instead of run_query_into() if new option pkeys_table_exists is on
- 12:51 PM Revision 2180: sql.py: mk_select(): Support joins with !=
- 12:45 PM Revision 2179: sql.py: mk_select(): Support only some join columns being join_using
- 12:40 PM Revision 2178: sql.py: put_table(): Renamed in_joins to insert_joins and joins to select_joins for clarity
- 12:21 PM Revision 2177: db_xml.py: put_table(): Support children with fkeys to parent
- 12:11 PM Revision 2176: sql.py: mk_select(): Make tuple optional for None literal values
05/13/2012
- 02:05 PM Revision 2175: sql.py: put_table(): Removed "SELECT statement missing a WHERE, LIMIT, or OFFSET clause" warnings
- 02:02 PM Revision 2174: bin/map: by_col: row_ct = 0 because it's unknown for now
- 02:00 PM Revision 2173: mk_select(): Support join conditions with literal values
- 01:42 PM Revision 2172: sql.py: mk_insert_select(): embeddable: Don't cache function_query because function def could change and then change back
- 01:35 PM Revision 2171: sql.py: with_savepoint(): Renamed savepoints to have "level" prefix, since the # indicates the level #
- 01:32 PM Revision 2170: sql.py: get_cur_query(): Also accept input params to combine with input_query, and pass input params when get_cur_query() is called
- 01:26 PM Revision 2169: sql.py: DbConn.run_query(): Pass input query to get_cur_query()
- 01:19 PM Revision 2168: sql.py: get_cur_query() and _add_cursor_info(): Support input_query param that will be used if the raw query is None. Pass input_query in DbConn.execute().
- 01:09 PM Revision 2167: sql.py: DbConn.run_query(): Check that query != None
- 01:05 PM Revision 2166: bin/map: out_is_db: Only rollback() and close() out_db if it was connected
- 01:04 PM Revision 2165: sql.py: DbConn: Added connected()
- 01:01 PM Revision 2164: sql.py: Wrapped calls to get_cur_query() that are used as strings in str(), because get_cur_query() can return None
- 12:57 PM Revision 2163: sql.py: next_version(): Versions start from 1, because first existing name was version 0
- 12:55 PM Revision 2162: put_table(): Use short name for temp_suffix now that version # will be added if needed
- 12:51 PM Revision 2161: sql.py: mk_select(): Parse join columns for literal values and table-scoped names as well
- 11:54 AM Revision 2160: mappings/DwC2-VegBIEN.specimens.csv: establishmentMeans: Call _toGrowthform on growthform
- 11:53 AM Revision 2159: schemas/vegbien.sql: Added _toGrowthform
- 11:19 AM Revision 2158: sql.py: put_table(): Changed temp_prefix to a suffix so main name won't be removed if name is truncated
- 11:14 AM Revision 2157: sql.py: mk_select(): fields: Support columns with tables. Changed syntax for literal values so that it wouldn't conflict with new syntax for columns with tables.
- 11:08 AM Revision 2156: iters.py: flatten(): If not an iterable, just return the value
- 10:32 AM Revision 2155: sql.py: put_table(): Pass in_pkeys and out_pkeys to run_query_into() by ref so they will be updated if the table names are changed
- 10:28 AM Revision 2154: sql.py: put_table(): Pass pkeys to run_query_into() by ref so it will be updated if the table name is changed
- 10:15 AM Revision 2153: sql.py: run_query_into(): If CREATE TABLE AS generates a DuplicateTableException, rename the table with a version # prepended
- 10:08 AM Revision 2152: sql.py: run_query_into(): Made into param a reference so that the function can change it, and renamed it to into_ref
- 09:36 AM Revision 2151: sql.py: run_query_into(): Made into param a reference so that the function can change it, and renamed it to into_ref
- 09:11 AM Revision 2150: sql.py: put_table(): If DuplicateKeyException: run_query_into() recoverably, so that DB errors such as DuplicateTableException will be parsed
- 09:07 AM Revision 2149: sql.py: Removed no-longer-needed try_insert()
- 09:05 AM Revision 2148: sql.py: Merged with_parsed_errors() into run_query() so all recoverable queries would automatically benefit from DB error message parsing. DbConn: Moved _add_cursor_info() to DbCursor.execute().
- 07:45 AM Revision 2147: sql.py: with_parsed_errors(): Raise DuplicateTableException for "relation already exists" errors instead of "table name specified more than once" errors
- 07:43 AM Revision 2146: sql.py: run_query_into(): Removed "DROP TABLE IF EXISTS" because sometimes when there are collisions in the temp table names, the code actually uses both "copies" of the temp table. Eventually, this situation will be resolved by adding a counter to the temp table name.
- 07:26 AM Revision 2145: sql.py: Cleaned up DbException's and subclasses' messages
- 07:26 AM Revision 2144: exc.py: ExceptionWithCause: Added cause_newline option to put the cause on its own line instead of on the message line
- 07:10 AM Revision 2143: sql.py: with_parsed_errors(): Also parse "table name specified more than once" errors as DuplicateTableExceptions
- 06:56 AM Revision 2142: sql.py: put_table(): Handle DuplicateKeyExceptions by running a select query on the unique constraint columns
- 06:14 AM Revision 2141: sql.py: mk_select(): Support tuples of tables, not just lists
- 05:29 AM Revision 2140: sql.py: with_parsed_errors(): Support table names that start with "_"
- 05:20 AM Revision 2139: sql.py: DbConn: Added with_savepoint(). with_savepoint(): Use new DbConn.with_savepoint().
- 04:13 AM Revision 2138: schemas/functions.sql: Added _toBool
- 04:12 AM Revision 2137: mappings/DwC2-VegBIEN.specimens.csv: establishmentMeans: Use _toBool on iscultivated, isnative
- 04:11 AM Revision 2136: schemas/functions.sql: Added _toBool
- 04:01 AM Revision 2135: schemas/functions.sql: Made trigger functions IMMUTABLE since they do not modify other tables
- 03:51 AM Revision 2134: sql.py: put_table(): Added support for putting just a window subset of the rows in the table. Removed "SELECT statement missing a WHERE, LIMIT, or OFFSET clause" warnings.
- 03:30 AM Revision 2133: sql.py: put_table(): Return the column where the pkeys are made available (the out_pkey) instead of taking it as an argument
- 03:20 AM Revision 2132: sql.py: put_table(): Get input pkeys corresponding to rows in insert and join together out_pkeys and in_pkeys into final pkeys table
- 01:04 AM Revision 2131: sql.py: put_table(): Fully support multiple in_tables, joined together using the main input table's pkey
- 01:02 AM Revision 2130: sql.py: mk_select(): joins: Fixed bug where USING-based joins did not have closing ")"
- 12:28 AM Revision 2129: db_xml.py: put_table(): Fixed bug where in_table was last in in_tables instead of first, causing it to be ignored by the current put_table() implementation, which only considers the first table name
- 12:17 AM Revision 2128: db_xml.py: put_table(): Fixed bug where pkeys_table returned by recursive call to put_table() needed to be prefixed with $ to be treated as an input column name rather than a literal value
Also available in: Atom