Project

General

Profile

Statistics
| Revision:

# Date Author Comment
2012 04/30/2012 03:40 AM Aaron Marcuse-Kubitza

xml_func.py: Added _collapse to collapse a subtree if the "value" element in it is NULL

2011 04/30/2012 01:44 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: definedvalue: Made definedvalue nullable so that each row of a datasource can have a uniform structure in VegBIEN, and to support reusing the same XML DOM tree for each row

2010 04/30/2012 01:11 AM Aaron Marcuse-Kubitza

xpath.py: Added is_xpath()

2009 04/30/2012 01:10 AM Aaron Marcuse-Kubitza

xml_dom.py: set_value(): If value is None and node is Element, remove value node entirely instead of setting node's value to None

2008 04/30/2012 01:02 AM Aaron Marcuse-Kubitza

xml_dom.py: Added value_node(). Use new value_node() in value() and set_value(). set_value(): If the node already has a value node, reuse it instead of appending a new value node.

2007 04/30/2012 12:35 AM Aaron Marcuse-Kubitza

xpath.py: put_obj(): Return the id_attr_node using get_1() because it should only be one node

2006 04/30/2012 12:30 AM Aaron Marcuse-Kubitza

xml_func.py: _simplifyPath: Also treat the elem as empty if the required node exists but is empty

2005 04/30/2012 12:04 AM Aaron Marcuse-Kubitza

db_xml.py: put_table(): Added part of put() code that should be common to both functions

2004 04/27/2012 06:16 PM Aaron Marcuse-Kubitza

xpath.py: put_obj(): Return a tuple of the inserted node and the id attr node

2003 04/27/2012 06:13 PM Aaron Marcuse-Kubitza

xpath.py: set_id(): When creating the id_path, use obj() (which deepcopy()s the entire path) because it prevents pointers w/o targets

2002 04/27/2012 06:05 PM Aaron Marcuse-Kubitza

xpath.py: set_id(): When creating the id_path, deepcopy() the id_elem because its keys will change in the main copy

2001 04/27/2012 05:47 PM Aaron Marcuse-Kubitza

xpath.py: set_id(): Return the path to the ID attr, which can be used to change the ID

2000 04/27/2012 05:25 PM Aaron Marcuse-Kubitza

xpath.py: put_obj(): Return the inserted node so it can be used to change the inserted value

1999 04/27/2012 05:08 PM Aaron Marcuse-Kubitza

main Makefile: Maps validation: Fixed bug where there would be infinite recursion with the Maps validation section before the Subdir forwarding section (it's unknown why this is necessary)

1998 04/26/2012 07:12 PM Aaron Marcuse-Kubitza

db_xml.py: put_table(): Added commit param to specify whether to commit after each query

1997 04/26/2012 06:55 PM Aaron Marcuse-Kubitza

bin/map: in_is_db: by_col: Use new put_table() (defined but not implemented yet)

1996 04/26/2012 06:54 PM Aaron Marcuse-Kubitza

db_xml.py: Added put_table() (without implementation)

1995 04/26/2012 06:52 PM Aaron Marcuse-Kubitza

xml_func.py: strip(): Remove _ignore XML funcs completely instead of replacing them with their values

1994 04/26/2012 06:26 PM Aaron Marcuse-Kubitza

bin/map: in_is_db: by_col: Prefix each input column name by "$"

1993 04/26/2012 06:11 PM Aaron Marcuse-Kubitza

bin/map: in_is_db: by_col: Strip off XML functions

1992 04/26/2012 06:09 PM Aaron Marcuse-Kubitza

xml_func.py: Added strip(). pop_value(): Support custom name of value param.

1991 04/26/2012 05:44 PM Aaron Marcuse-Kubitza

bin/map: in_is_db: by_col: Create XML tree of sample row, with the input column names as the values. This tree will guide the sequencing and creation of the column-based queries.

1990 04/26/2012 05:43 PM Aaron Marcuse-Kubitza

input.Makefile: use_staged env var: defaults to on if by_col is on

1989 04/26/2012 05:00 PM Aaron Marcuse-Kubitza

bin/map: Only turn on by_col optimization if mapping to same DB, rather than requiring each place that checks by_col to also check whether mapping to same DB

1988 04/24/2012 06:32 PM Aaron Marcuse-Kubitza

input.Makefile: Testing: Don't abort tester if only staging test fails, in case staging table missing

1987 04/24/2012 06:25 PM Aaron Marcuse-Kubitza

input.Makefile: Testing: When cleaning up test outputs, remove everything that doesn't end in .ref

1986 04/24/2012 06:11 PM Aaron Marcuse-Kubitza

input.Makefile: Testing: Added test/import.%.staging.out test to test the staging tables. Sources: cat: Updated Usage comment to include the "inputs/<datasrc>/" prefix the user would need to add when running make.

1985 04/24/2012 05:33 PM Aaron Marcuse-Kubitza

bin/map: Fixed bug where mapping to same DB wouldn't work because by-column optimization wasn't implemented yet, by turning it off by default and allowing it to be enabled with an env var

1984 04/24/2012 05:25 PM Aaron Marcuse-Kubitza

bin/map: DB inputs: Use by-column optimization if mapping to same DB (with skeleton code for optimization's implementation)

1983 04/24/2012 05:12 PM Aaron Marcuse-Kubitza

input.Makefile: Mapping: Use the staging tables instead of any flat files if use_staged is specified

1982 04/24/2012 05:10 PM Aaron Marcuse-Kubitza

bin/map: Support custom schema name. Support input table/schema override via env vars, in case the map spreadsheet was written for a different input format.

1981 04/24/2012 05:01 PM Aaron Marcuse-Kubitza

sql.py: qual_name(): Fixed bugs where esc_name() nested func couldn't have same name as outer func, and esc_name() needed to be invoked without the module name because it's in the same module. select(): Support already-escaped table names.

1980 04/24/2012 04:16 PM Aaron Marcuse-Kubitza

main Makefile: $(psqlAsAdmin): Tell sudo to preserve env vars so PGOPTIONS is passed to psql

1979 04/24/2012 03:33 PM Aaron Marcuse-Kubitza

root map: Fill in defaults for inputs from VegBIEN, as well as outputs to it

1978 04/24/2012 02:59 PM Aaron Marcuse-Kubitza

disown_all: Updated to use main function, local vars, $self, etc. like other bash scripts run using "."

1977 04/24/2012 02:55 PM Aaron Marcuse-Kubitza

vegbien_dest: Fixed bug where it would give a usage error if run from a makefile rule, because the BASH_LINENO would be 0, by also checking if ${BASH_ARGV0} is ${BASH_SOURCE0}

1976 04/24/2012 02:28 PM Aaron Marcuse-Kubitza

postgres_vegbien: Fixed bug where interpreter did not match vegbien_dest's new required interpreter of /bin/bash

1975 04/24/2012 02:23 PM Aaron Marcuse-Kubitza

vegbien_dest: Changed interpreter to /bin/bash. Removed comment that it requires var bien_password.

1974 04/24/2012 02:20 PM Aaron Marcuse-Kubitza

postgres_vegbien: Removed no longer needed retrieval of bien_password

1973 04/24/2012 02:20 PM Aaron Marcuse-Kubitza

vegbien_dest: Get bien_password by searching relative to $self, which we now have a way to get in a bash script (${BASH_SOURCE0}), rather than requiring the caller to set it. Provide usage error if run without initial ".".

1972 04/24/2012 02:12 PM Aaron Marcuse-Kubitza

input.Makefile: Staging tables: import/install-%: Use new quiet option to determine whether to tee output to terminal. Don't use log option because that's always set to true except in test mode, which doesn't apply to installs.

1971 04/24/2012 02:12 PM Aaron Marcuse-Kubitza

input.Makefile: Staging tables: import/install-%: Use new quiet option to determine whether to tee output to terminal. Don't use log option because that's always set to true except in test mode, which doesn't apply to installs.

1970 04/24/2012 01:56 PM Aaron Marcuse-Kubitza

main Makefile: PostgreSQL: Edit /etc/phppgadmin/apache.conf to replace "deny from all" with "allow from all", instead of uncommenting an "allow from all" that may not be there

1969 04/24/2012 01:35 PM Aaron Marcuse-Kubitza

input.Makefile: Sources: Fixed bug where cat was defined before $(tables), by moving Sources after Existing maps discovery and putting just $(inputFiles) and $(dbExport) from Sources at the beginning of Existing maps discovery

1968 04/24/2012 01:05 PM Aaron Marcuse-Kubitza

sql.py: Made truncate(), tables(), empty_db() schema-aware. Added qual_name(). tables(): Added option to filter tables by a LIKE pattern.

1967 04/24/2012 12:34 PM Aaron Marcuse-Kubitza

main Makefile: VegBIEN DB: Install public schema in a separate step, so that it can be dropped without dropping the entire DB (which also contains staging tables that shouldn't be dropped when there is a schema change). Added schemas/install, schemas/uninstall, implicit schemas/reinstall to manage the public schema separately from the rest of the DB. Moved Subdir forwarding to the bottom so overridden targets are not forwarded. README.TXT: Since `make reinstall_db` would drop the entire DB, tell user to run new `make schemas/reinstall` instead to reinstall (main) DB from schema.

1966 04/24/2012 12:30 PM Aaron Marcuse-Kubitza

schemas/postgresql.Mac.conf: Set unix_socket_directory to the new dir it seems to be using, which is now /tmp

1965 04/24/2012 11:43 AM Aaron Marcuse-Kubitza

csv2db: Fixed bug where extra columns were not truncated in INSERT mode. Replace empty column names with the column # to avoid errors with CSVs that have trailing ","s, etc.

1964 04/24/2012 11:41 AM Aaron Marcuse-Kubitza

streams.py: StreamIter: Define readline() as a separate method so it can be overridden, and all calls to self.next() will use the overridden readline(). This fixes a bug in ProgressInputStream where incremental counts would not be displayed and it would end with "not all input read" if the StreamIter interface was used instead of readline().

1963 04/23/2012 09:57 PM Aaron Marcuse-Kubitza

csv2db: Fall back to manually inserting each row (autodetecting the encoding for each field) if COPY FROM doesn't work

1962 04/23/2012 09:56 PM Aaron Marcuse-Kubitza

streams.py: FilterStream: Inherit from StreamIter so that all descendants automatically have StreamIter functionality

1961 04/23/2012 09:42 PM Aaron Marcuse-Kubitza

sql.py: insert(): Support using the default value for columns designated with the special value sql.default

1960 04/23/2012 09:21 PM Aaron Marcuse-Kubitza

sql.py: insert(): Support rows that are just a list of values, with no columns. Support already-escaped table names.

1959 04/23/2012 08:54 PM Aaron Marcuse-Kubitza

strings.py: Added contains_any()

1958 04/23/2012 08:54 PM Aaron Marcuse-Kubitza

csvs.py: reader_and_header(): Use make_reader()

1957 04/23/2012 08:07 PM Aaron Marcuse-Kubitza

Added reinstall_all to reinstall all inputs at once

1956 04/23/2012 08:06 PM Aaron Marcuse-Kubitza

with_all: Documented that it must be run from the root svn directory

1955 04/23/2012 08:05 PM Aaron Marcuse-Kubitza

input.Makefile: Staging tables: import/install-%: Only install staging table if input contains only CSV sources. Changed $(isXml) to $(isCsv) (negated) everywhere because rules almost always only run something if input contains only CSV sources, rather than if input contains XML sources.

1954 04/23/2012 07:21 PM Aaron Marcuse-Kubitza

input.Makefile: Staging tables: import/install-%: Output load status to log file if log option is set

1953 04/23/2012 07:00 PM Aaron Marcuse-Kubitza

Scripts that are meant to be run in the calling shell: Fixed bug where running the script inside another script would make the script think it was being run as a program, and abort with a usage error

1952 04/23/2012 06:56 PM Aaron Marcuse-Kubitza

Scripts that are meant to be run in the calling shell: Fixed bug where running the script as a program (without initial ".") wouldn't be able to call return in something that was not a function. Converted all code to a <script_name>_main method so that return would work properly again. Converted all variables to local variables.

1951 04/23/2012 06:38 PM Aaron Marcuse-Kubitza

env_password: return instead of exit if password not yet stored, in case user is running it from a shell without the initial "-" argument. (This would be the case if the user is just testing out the script, instead of using a command that env_password directs them to run.)

1950 04/23/2012 05:43 PM Aaron Marcuse-Kubitza

env_password: Use ${BASH_SOURCE0} for $self and $self for $0. return instead of exit on usage error in case user is running it from a shell.

1949 04/23/2012 05:36 PM Aaron Marcuse-Kubitza

stop_imports: Use ${BASH_SOURCE0} for $self and $self for $0

1948 04/23/2012 05:36 PM Aaron Marcuse-Kubitza

import_all: Use new with_all. Use ${BASH_SOURCE0} for $self and $self for $0.

1947 04/23/2012 05:34 PM Aaron Marcuse-Kubitza

Added with_all to run a make target on all inputs at once

1946 04/23/2012 05:05 PM Aaron Marcuse-Kubitza

Made row #s 1-based to the user to match up with the staging table row #s

1945 04/23/2012 04:59 PM Aaron Marcuse-Kubitza

bin/map: Fixed bug where limit passed to sql.select() was end instead of the # rows, causing extra rows to be fetched when start > 0. Documented that row #s start with 0.

1944 04/23/2012 04:19 PM Aaron Marcuse-Kubitza

Removed no longer needed csv2ddl

1943 04/23/2012 04:19 PM Aaron Marcuse-Kubitza

input.Makefile: Staging tables: import/install-%: Use new csv2db instead of csv2ddl/$(psqlAsBien), because it handles translating encodings properly

1942 04/23/2012 04:14 PM Aaron Marcuse-Kubitza

Added csv2db to load a command's CSV output stream into a PostgreSQL table

1941 04/21/2012 09:32 PM Aaron Marcuse-Kubitza

schemas/postgresql.Mac.conf: Set unix_socket_directory to the appropriate Mac OS X dir, since otherwise, the socket is apparently not created and `make reinstall_db` doesn't work

1940 04/21/2012 09:30 PM Aaron Marcuse-Kubitza

main Makefile: VegBIEN DB: db: Set LC_COLLATE and LC_CTYPE explicitly, to make it easier to change them

1939 04/21/2012 09:29 PM Aaron Marcuse-Kubitza

Added ProgressInputStream

1938 04/21/2012 09:28 PM Aaron Marcuse-Kubitza

exc.py: print_ex(): Added plain option to leave out traceback

1937 04/21/2012 06:48 PM Aaron Marcuse-Kubitza

main Makefile: VegBIEN DB: db: Use template0 to allow encodings other than UTF-8. Because template0 doesn't have plpgsql on PostgreSQL before 9.x, add "CREATE PROCEDURAL LANGUAGE plpgsql;" manually in schemas/vegbien.sql.make, and filter it back out on PostgreSQL after 9.x using db_dump_localize.

1936 04/21/2012 06:39 PM Aaron Marcuse-Kubitza

PostgreSQL-MySQL.csv: Remove "CREATE PROCEDURAL LANGUAGE" statements

1935 04/21/2012 06:36 PM Aaron Marcuse-Kubitza

Added db_dump_localize to translate a PostgreSQL DB dump for the local server's version

1934 04/21/2012 06:32 PM Aaron Marcuse-Kubitza

Added db_dump_localize to translate a PostgreSQL DB dump for the local server's version

1933 04/21/2012 03:42 PM Aaron Marcuse-Kubitza

vegbien_dest: Added option to override the prefix of the created vars

1932 04/21/2012 03:35 PM Aaron Marcuse-Kubitza

schemas/vegbien.sql.make: Fixed bug where data sources' schemas were also exported by exporting only the public schema. Note that this also removes the "CREATE OR REPLACE PROCEDURAL LANGUAGE plpgsql" statement, so that it doesn't have to be filtered out with `grep -v`.

1931 04/21/2012 03:19 PM Aaron Marcuse-Kubitza

input.Makefile: input.Makefile: Use `$(catSrcs)|` instead of $(withCatSrcs) where possible

1930 04/21/2012 03:00 PM Aaron Marcuse-Kubitza

sql.py: pkey(): Fixed bug where results were not being cached because the rows hadn't been explicitly fetched, by having DbConn.DbCursor.execute() fetch all rows if the rowcount is 0 and it's not an insert statement. DbConn.DbCursor: Made _is_insert an attribute rather than a method, which is set as soon as the query is known. Added consume_rows(). Moved Result retrieval section above Database connections because it's used by DbConn.

1929 04/21/2012 02:28 PM Aaron Marcuse-Kubitza

sql.py: pkey(): Fixed bug where queries were not being cached. Use select() instead of run_query() so that caching is automatically turned on and table names are automatically escaped.

1928 04/21/2012 01:37 PM Aaron Marcuse-Kubitza

streams.py: Added LineCountInputStream, which is faster than LineCountStream for input streams. Added InputStreamsOnlyException and raise it in all *InputStream classes' write() methods.

1927 04/21/2012 01:22 PM Aaron Marcuse-Kubitza

sql.py: DbConn: For non-cacheable queries, use a plain cursor() instead of a DbCursor to avoid the overhead of saving the result and wrapping the cursor

1926 04/20/2012 05:20 PM Aaron Marcuse-Kubitza

Moved db_config_names from bin/map to sql.py so it can be used by other scripts as well

1925 04/20/2012 04:52 PM Aaron Marcuse-Kubitza

csv2ddl: Also print a COPY FROM statement

1924 04/20/2012 04:47 PM Aaron Marcuse-Kubitza

input.Makefile: Fixed bug where input type was considered to be different things if both $(inputFiles) and $(dbExport) are non-empty. Now, $(inputFiles) takes precedence so that the presence of any input files will cause a DB dump to be ignored. This ensures that a (slower) input DB is not used over a (faster) flat file.

1923 04/20/2012 04:21 PM Aaron Marcuse-Kubitza

csvs.py: stream_info(): Added parse_header option. reader_and_header(): Use stream_info()'s new parse_header option.

1922 04/20/2012 03:53 PM Aaron Marcuse-Kubitza

csv2ddl: Renamed schema name env var from datasrc to schema to reflect what it is, and to make the script general beyond importing inputs

1921 04/20/2012 03:32 PM Aaron Marcuse-Kubitza

input.Makefile: Moved Installation, Staging tables after Existing maps discovery because they depend on it. Staging tables: Create a staging table for each table a map spreadsheet is available for. Put double quotes around the schema name so its case is preserved.

1920 04/20/2012 03:29 PM Aaron Marcuse-Kubitza

Added csv2ddl to make a PostgreSQL CREATE TABLE statement from a CSV header

1919 04/20/2012 03:28 PM Aaron Marcuse-Kubitza

sql.py: Input validation: Moved section after Database connections because some of its functions require a connection. Added esc_name_by_module() and esc_name_by_engine(), and use esc_name_by_module() in esc_name().

1918 04/20/2012 02:18 PM Aaron Marcuse-Kubitza

input.Makefile: Installation: Create a schema for the datasource in VegBIEN as part of the installation process. This will be used to hold staging tables.

1917 04/20/2012 01:57 PM Aaron Marcuse-Kubitza

input.Makefile: Changed install, uninstall to depend on src/install, src/uninstall targets, which in turn depend on db, rm_db. This will allow us to add additional install actions for all input types.

1916 04/19/2012 07:17 PM Aaron Marcuse-Kubitza

sql.py: DbConn: Cache the constructed CacheCursor itself, rather than the dict that's used to create it

1915 04/19/2012 07:06 PM Aaron Marcuse-Kubitza

sql.py: pkey(): Changed to use the connection-wide caching mechanism rather than its own custom cache. DbConn.__getstate__(): Don't pickle the debug callback.

1914 04/19/2012 07:00 PM Aaron Marcuse-Kubitza

sql.py: DbConn: Added is_cached(). run_query(): Use new DbConn.is_cached() to avoid creating a savepoint if the query is cached.

1913 04/19/2012 06:52 PM Aaron Marcuse-Kubitza

sql.py: DbConn: Also cache cursor.description