Project

General

Profile

Statistics
| Revision:

# Date Author Comment
2904 06/15/2012 05:02 AM Aaron Marcuse-Kubitza

sql.py: put_table(): Removed locking output tables to prevent concurrent duplicate keys because that is now done automatically by mk_select()

2903 06/15/2012 05:01 AM Aaron Marcuse-Kubitza

sql.py: mk_select(): Filtering on no match: Lock the joined table in EXCLUSIVE mode to prevent concurrent duplicate keys when used with INSERT SELECT

2902 06/15/2012 04:59 AM Aaron Marcuse-Kubitza

sql_gen.py: Added underlying_table() and use it in underlying_col()

2901 06/15/2012 04:39 AM Aaron Marcuse-Kubitza

main Makefile: schemas/rotate: Fixed bug where needed to run schemas/public/install, not full schemas/install, after renaming public schema

2900 06/15/2012 04:32 AM Aaron Marcuse-Kubitza

sql.py: put_table(): Lock output tables to prevent concurrent duplicate keys

2899 06/15/2012 04:31 AM Aaron Marcuse-Kubitza

sql.py: Added lock_table()

2898 06/15/2012 03:53 AM Aaron Marcuse-Kubitza

bin/map: connect_db(): Only use autocommit mode if verbosity > 3, to avoid accidentally activating it if you want debug output in normal import mode

2897 06/15/2012 03:45 AM Aaron Marcuse-Kubitza

bin/map: connect_db(): Only use autocommit mode if verbosity > 2, because it causes the intermediate tables to be created as permanent tables, which you don't want unless you're actually debugging (verbosity = 2 is normal for column-based import)

2896 06/15/2012 03:25 AM Aaron Marcuse-Kubitza

sql.py: put_table(): remove_all_rows(): Changed log message to "Ignoring all rows" because NULL is not necessarily the pkey value that will be returned for the rows

2895 06/15/2012 03:17 AM Aaron Marcuse-Kubitza

sql.py: put_table(): Don't add index on columns that will have values filtered out, because indexes have already been added on all columns in the iteration's input table by flatten()

2894 06/15/2012 03:12 AM Aaron Marcuse-Kubitza

sql.py: DbConn._db(): Setting serializable isolation level: Always set this (if self.serializable is set), even in autocommit mode, because autocommit mode is implemented by manual commits in the DbConn wrapper object rather than using the underlying connection's autocommit mode (which does not allow setting the isolation level)

2893 06/15/2012 03:08 AM Aaron Marcuse-Kubitza

sql.py: DbConn._db(): Setting search_path: Use `SET search_path` and `SHOW search_path` instead of combining the old and new search_paths in SQL itself using `SELECT set_config('search_path', ...)`

2892 06/15/2012 02:47 AM Aaron Marcuse-Kubitza

csv2db: ProgressInputStream: Use default progress message 'Read %d line(s)' because there is not necessarily one CSV row per line, due to embedded newlines

2891 06/15/2012 01:47 AM Aaron Marcuse-Kubitza

input.Makefile: Staging tables: import/install-%: Only output to the log file if log option is non-empty (which it is by default)

2890 06/15/2012 01:46 AM Aaron Marcuse-Kubitza

csv2db: Support reinstalling just the errors table using new errors_table_only option

2889 06/15/2012 01:45 AM Aaron Marcuse-Kubitza

sql.py: Added drop_table()

2888 06/15/2012 01:20 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: method: Changed indexes to use `COALESCE` to match what sql_gen now does

2887 06/15/2012 01:16 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: specimenreplicate: Added indexes using COALESCE to match what sql_gen does

2886 06/15/2012 01:12 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: locationevent: Added indexes using COALESCE to match what sql_gen does

2885 06/15/2012 12:57 AM Aaron Marcuse-Kubitza

schemas/vegbien.ERD.mwb: Synced with schema

2884 06/15/2012 12:54 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: party: Changed indexes to use `COALESCE` to match what sql_gen now does

2883 06/15/2012 12:38 AM Aaron Marcuse-Kubitza

Wrap sys.stderr.write() calls in strings.to_raw_str() to avoid UnicodeEncodeErrors when stderr is to a file and the default encoding is ASCII

2882 06/15/2012 12:37 AM Aaron Marcuse-Kubitza

strings.py: Added to_raw_str()

2881 06/15/2012 12:12 AM Aaron Marcuse-Kubitza

bin/map: When logging the row # being processed, add 1 because row # is interally 0-based, but 1-based to the user

2880 06/15/2012 12:05 AM Aaron Marcuse-Kubitza

bin/map: Log the row # being processed with level=1.1 so that the user can see a status report if desired

2879 06/14/2012 11:35 PM Aaron Marcuse-Kubitza

exc.py: str_(): Fixed bug where UnicodeEncodeError would be raised when msg contains non-ASCII chars, by wrapping e.args0 in strings.ustr()

2878 06/14/2012 11:23 PM Aaron Marcuse-Kubitza

exc.py: print_ex(): Wrap msg in strings.to_unicode() to try to avoid UnicodeEncodeError when msg contains non-ASCII chars

2877 06/14/2012 11:20 PM Aaron Marcuse-Kubitza

sql.py: create_table(): Don't set pkey.nullable to False because the caller should make sure the pkey has the appropriate type

2876 06/14/2012 11:20 PM Aaron Marcuse-Kubitza

csv2db: Use sql_gen.TypedCol.nullable instead of manually adding 'NOT NULL' to the type. Ensure that pkeys are properly NOT NULL.

2875 06/14/2012 11:15 PM Aaron Marcuse-Kubitza

csv2db: Adding indexes: Create plain indexes using ensure_not_null=False because the indexes will primarily be used by the user to search for specific values, rather than by the mapping script which uses the ensure_not_null

2874 06/14/2012 11:13 PM Aaron Marcuse-Kubitza

sql.py: DbConn.col_info(): Run query with log_level=4 because it gathers information about database structure, and should have the same log_level as other queries that do that

2873 06/14/2012 11:09 PM Aaron Marcuse-Kubitza

csv2db: Adding indexes: Fixed bug where col.to_Col() could not be used because sql.add_index() does not support name-only columns (plain strings are OK, though)

2872 06/14/2012 11:00 PM Aaron Marcuse-Kubitza

sql.py: create_table(): has_pkey: Use new TypedCol.constraints to store 'PRIMARY KEY'

2871 06/14/2012 10:59 PM Aaron Marcuse-Kubitza

sql_gen.py: TypedCol: Added constraints instance var

2870 06/14/2012 10:38 PM Aaron Marcuse-Kubitza

sql_gen.py: EnsureNotNull: Made coalesce() all uppercase to match how pg_dump spells it

2869 06/14/2012 10:36 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: namedplace: Fixed bug where parent_id needed to be included in UNIQUE CONSTRAINT (now UNIQUE INDEX), since there can be more than one e.g. city of the same name if they are in different states

2868 06/14/2012 10:32 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql: plantname: Changed indexes to use `COALESCE` to match what sql_gen now does

2867 06/14/2012 10:22 PM Aaron Marcuse-Kubitza

schemas/py_functions.sql: _dateRangeStart, _dateRangeEnd: Changed indexes to use `COALESCE` to match what sql_gen now does

2866 06/14/2012 10:19 PM Aaron Marcuse-Kubitza

schemas/py_functions.sql: _namePart: Changed indexes to use `COALESCE` to match what sql_gen now does

2865 06/14/2012 10:16 PM Aaron Marcuse-Kubitza

schemas/functions.sql: _nullIf: Changed indexes to use `COALESCE` to match what sql_gen now does

2864 06/14/2012 10:15 PM Aaron Marcuse-Kubitza

schemas/functions.sql: _nullIf: Require a non-NULL null-equivalent value

2863 06/14/2012 10:12 PM Aaron Marcuse-Kubitza

schemas/functions.sql: _label: Changed indexes to use `COALESCE` to match what sql_gen now does

2862 06/14/2012 10:09 PM Aaron Marcuse-Kubitza

schemas/functions.sql: _label: Require a non-NULL label

2861 06/14/2012 10:04 PM Aaron Marcuse-Kubitza

sql_gen.py: null_sentinels: Removed types where a sentinel doesn't make sense (unknown types, boolean) because types with no sentinel are now handled gracefully by users of ensure_not_null()

2860 06/14/2012 10:00 PM Aaron Marcuse-Kubitza

sql.py: add_index(): ensure_not_null: Handle unknown types gracefully

2859 06/14/2012 09:52 PM Aaron Marcuse-Kubitza

sql_gen.py: MockDb: Added col_info()

2858 06/14/2012 09:48 PM Aaron Marcuse-Kubitza

sql_gen.py: Use as_*() functions where the auto-wrapping was previously done manually

2857 06/14/2012 09:44 PM Aaron Marcuse-Kubitza

sql_gen.py: CompareCond.to_str(): Use ensure_not_null()'s new type_ param to apply same function to both sides but not if the right side is already NOT NULL

2856 06/14/2012 09:26 PM Aaron Marcuse-Kubitza

sql_gen.py: null_sentinels: Added value for character varying type

2855 06/14/2012 08:45 PM Aaron Marcuse-Kubitza

sql_gen.py: ensure_not_null(): Support non-column inputs if type_ is set

2854 06/14/2012 08:32 PM Aaron Marcuse-Kubitza

sql_gen.py: null_sentinels: Added value for USER-DEFINED type

2853 06/14/2012 08:31 PM Aaron Marcuse-Kubitza

sql.py: mk_select(): Joins: Filtering on no match: Use '~=' sql_gen.CompareCond operator so that IS NULL is always used, regardless of the not-null column's nullability

2852 06/14/2012 08:24 PM Aaron Marcuse-Kubitza

sql_gen.py: null_sentinels: Added value for boolean type

2851 06/14/2012 08:21 PM Aaron Marcuse-Kubitza

sql_gen.py: ensure_not_null(): Added type_ param to override the underlying column's type

2850 06/14/2012 08:09 PM Aaron Marcuse-Kubitza

sql_gen.py: EnsureNotNull: Take a type param instead of a null param so that the EnsureNotNull object stores the underlying column's type

2849 06/14/2012 08:00 PM Aaron Marcuse-Kubitza

sql_gen.py: underlying_col(): Support non-Col inputs

2848 06/14/2012 07:42 PM Aaron Marcuse-Kubitza

sql_gen.py: EnsureNotNull: Removed default value for null param to remind user that default value depends on value's type and will not always be a string

2847 06/14/2012 07:39 PM Aaron Marcuse-Kubitza

sql.py: add_index(): Added ensure_not_null param to disable the ensure_not_null functionality to force a plain index

2846 06/14/2012 07:14 PM Aaron Marcuse-Kubitza

sql.py: flatten(): Add indexes on the created table so its columns can be used in an O(n) merge join

2845 06/14/2012 07:12 PM Aaron Marcuse-Kubitza

sql_gen.py: null_sentinels: Added value for integer type

2844 06/14/2012 06:52 PM Aaron Marcuse-Kubitza

sql_gen.py: CompareCond.to_str(): Always wrap the left-side column if it's nullable. Wrap the right-side value if the left side was wrapped, rather than if both the left and right side are nullable. This causes coalesce() indexes to be used to look up NULL values using the value NULL gets coalesced to, rather than doing a sequential scan.

2843 06/14/2012 06:43 PM Aaron Marcuse-Kubitza

sql_gen.py: Run truncate() on all identifiers so that literal-string-based lookups for an identifier (such as in db.col_info()) don't use the untruncated value

2842 06/14/2012 06:40 PM Aaron Marcuse-Kubitza

sql_gen.py: Added truncate()

2841 06/14/2012 06:40 PM Aaron Marcuse-Kubitza

sql.py: put_table(): Resolving default value column: Fixed bug where default value of None was used as a key for mapping, even though this is an invalid Col name

2840 06/14/2012 05:30 PM Aaron Marcuse-Kubitza

sql_gen.py: ensure_not_null(): If input column cannot be ensured to be NULL, pass any raised exception through rather than suppressing it and leaving the column in a nullable state

2839 06/13/2012 11:11 PM Aaron Marcuse-Kubitza

schemas/functions.sql: _merge: Changed indexes to use `COALESCE` to match what sql_gen now does

2838 06/13/2012 11:08 PM Aaron Marcuse-Kubitza

schemas/functions.sql: _alt: Changed indexes to use `COALESCE` to match what sql_gen now does

2837 06/13/2012 10:59 PM Aaron Marcuse-Kubitza

sql_gen.py: CompareCond.to_str(): Handle nullable columns using ensure_not_null()

2836 06/13/2012 10:46 PM Aaron Marcuse-Kubitza

sql_gen.py: ensure_not_null(): Raise NoUnderlyingTableException if can't ensure not null for that reason

2835 06/13/2012 10:20 PM Aaron Marcuse-Kubitza

sql_gen.py: is_underlying_table(): Support non-Table inputs

2834 06/13/2012 10:15 PM Aaron Marcuse-Kubitza

sql_gen.py: NamedValues: Call set_cols_table() with the created table, not just the name, so that is_underlying_table() works properly

2833 06/13/2012 09:49 PM Aaron Marcuse-Kubitza

sql_gen.py: underlying_col(): If no underlying table, raise NoUnderlyingTableException

2832 06/13/2012 09:45 PM Aaron Marcuse-Kubitza

sql_gen.py: Added is_underlying_table()

2831 06/13/2012 09:34 PM Aaron Marcuse-Kubitza

sql_gen.py: ensure_not_null(): Call underlying_col() on the column to remove all renamings

2830 06/13/2012 09:33 PM Aaron Marcuse-Kubitza

sql_gen.py: Added underlying_col()

2829 06/13/2012 09:24 PM Aaron Marcuse-Kubitza

sql_gen.py: Join.to_str(): join(): Removed no longer needed `*_table = *_table.to_Table()`

2828 06/13/2012 09:22 PM Aaron Marcuse-Kubitza

sql_gen.py: Col: Support Table objects that are not just names, by calling `.to_Table()` on the table before stringifying it

2827 06/13/2012 09:22 PM Aaron Marcuse-Kubitza

sql_gen.py: Col: Support Table objects that are not just names, by calling `.to_Table()` on the table before stringifying it

2826 06/13/2012 08:59 PM Aaron Marcuse-Kubitza

sql_gen.py: ensure_not_null(): Added ignore_unknown_type param

2825 06/13/2012 08:55 PM Aaron Marcuse-Kubitza

sql_gen.py: CompareCond.to_str(): Put handling nullable columns as a separate step so it can be expanded

2824 06/13/2012 08:36 PM Aaron Marcuse-Kubitza

csv2db: Errors table: Removed no longer needed sql_gen.EnsureNotNull() because this is now added automatically

2823 06/13/2012 08:35 PM Aaron Marcuse-Kubitza

sql.py: add_index(): Handle nullable columns by using sql_gen.ensure_not_null() to automatically add a coalesce() wrapper where needed

2822 06/13/2012 08:33 PM Aaron Marcuse-Kubitza

sql_gen.py: Added ensure_not_null()

2821 06/13/2012 08:29 PM Aaron Marcuse-Kubitza

sql.py: DbConn.col_info(): Fixed bug where is_nullable needed to be cast to a boolean

2820 06/13/2012 08:06 PM Aaron Marcuse-Kubitza

sql.py: cast(): Support string column name inputs

2819 06/13/2012 07:12 PM Aaron Marcuse-Kubitza

sql.py: DbConn: Renamed col_default() to col_info() and have it return a sql_gen.TypedCol object containing all the TypedCol info about the column, not just the default value

2818 06/13/2012 06:55 PM Aaron Marcuse-Kubitza

sql_gen.py: TypedCol: Added default and nullable params

2817 06/13/2012 06:53 PM Aaron Marcuse-Kubitza

dicts.py: Import util after items that util depends on have been defined, to avoid unsatisfied circular dependency

2816 06/13/2012 06:30 PM Aaron Marcuse-Kubitza

sql.py: DbConn.col_default(): Pass the connection to sql_gen.as_Code() so it fixes the syntax on values returned by PostgreSQL

2815 06/13/2012 06:29 PM Aaron Marcuse-Kubitza

sql_gen.py: as_Code(): Added optional db param, which causes the function to run db.std_code() on the value to fix the syntax

2814 06/13/2012 06:27 PM Aaron Marcuse-Kubitza

sql.py: DbConn: Added std_code()

2813 06/13/2012 03:02 PM Aaron Marcuse-Kubitza

db_xml.py: Removed into_table_name() because this functionality is now handled by sql.into_table_name()

2812 06/13/2012 02:59 PM Aaron Marcuse-Kubitza

sql.py: into_table_name(): Also parse hierarchical tables (mappings with a rank column) using a special syntax

2811 06/13/2012 02:44 PM Aaron Marcuse-Kubitza

sql.py: put_table(): Fixed bug where distinct_on included columns that were not in the input table, and were thus incorrectly taken from the LEFT JOINed output table

2810 06/12/2012 10:32 PM Aaron Marcuse-Kubitza

sql.py: track_data_error(): Do nothing if cols are empty, because mk_track_data_error() requires at least one col. mk_track_data_error(): Assert that cols are not empty because VALUES clause requires at least one row.

2809 06/12/2012 10:26 PM Aaron Marcuse-Kubitza

bin/map: by_col: Pass on_error to db_xml.put_table() that calls ex_tracker.track()

2808 06/12/2012 10:14 PM Aaron Marcuse-Kubitza

sql.py: put_table(): No handler for exception: Pass exception to on_error() instead of raising a warning, so that error message can be formatted

2807 06/12/2012 10:08 PM Aaron Marcuse-Kubitza

db_xml.py: put_table(): Pass on_error to sql.put_table()

2806 06/12/2012 10:07 PM Aaron Marcuse-Kubitza

db_xml.py: put_table(): Take on_error param like row-based put()

2805 06/12/2012 10:06 PM Aaron Marcuse-Kubitza

sql.py: put_table(): Take on_error param