Project

General

Profile

Statistics
| Revision:

# Date Author Comment
6989 12/20/2012 01:23 PM Aaron Marcuse-Kubitza

mappings/VegCore-VegBIEN.csv: Mapped authorTaxonCode

6988 12/20/2012 01:12 PM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Regenerated from wiki

6987 12/20/2012 01:12 PM Aaron Marcuse-Kubitza

README.TXT: Maintenance: VegCore data dictionary: Added step to commit updated mappings/VegCore.csv

6986 12/20/2012 12:13 PM Aaron Marcuse-Kubitza

schemas/Makefile: %/publish: Fixed bug where commands were not being run transactionally, because --single-transaction requires `--file -` to work properly

6985 12/20/2012 11:36 AM Aaron Marcuse-Kubitza

input.Makefile: Editing import: Removed rotate because appending the current svn revision doesn't make sense, since this is not related to the revision used to import the datasource

6984 12/20/2012 11:34 AM Aaron Marcuse-Kubitza

input.Makefile: Editing import: Added rename/% and use it in rotate

6983 12/20/2012 11:21 AM Aaron Marcuse-Kubitza

inputs/import.stats.xls: Updated import times

6982 12/20/2012 11:21 AM Aaron Marcuse-Kubitza

schemas/Makefile: Use $* instead of $(@D) for clarity. $(@D) is only needed when the dir part of the target includes a prefix in addition to the % stem.

6981 12/20/2012 10:45 AM Aaron Marcuse-Kubitza

make_analytical_db: Automatically call export_analytical_db when finished

6980 12/20/2012 10:35 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: make_family_higher_plant_group(): Added `taxonepithet IS NOT NULL` filter, to allow make_analytical_db to proceed even when the NCBI import fails (leaving some nodes with rank = 'family' but no associated taxonepithet). The most recent NCBI import failed due to the search_path/DuplicateException bug resulting from the import schema and public being in the search_path together.

6979 12/20/2012 10:14 AM Aaron Marcuse-Kubitza

schemas/Makefile: Fixed bug where need `SHELL := /bin/bash` for \$(confirmRmPublicSchema) to work correctly

6978 12/20/2012 10:12 AM Aaron Marcuse-Kubitza

lib/common.Makefile: $(confirm): Added comment that this requires `SHELL := /bin/bash` to work correctly

6977 12/20/2012 10:09 AM Aaron Marcuse-Kubitza

import_all: after_import(): Added `make backups/vegbien.$version.backup/test`

6976 12/20/2012 10:05 AM Aaron Marcuse-Kubitza

sql.py: DbConn._db(): search_path: Don't append the existing search_path, because it usually includes the public schema, which is now different from the schema being imported into. This fixes a bug where sql.function_exists() would find public-schema functions in both the public schema and the import's schema because both were in the search_path, causing a DuplicateException "more than one function named ...". Note that the elements of the existing search_path are no longer needed now that vegbien_dest's $schemas includes $public. Also note that if an instance of DbConn does not specify the schemas param, the existing search_path will be left as-is rather than overwritten with an empty list.

6975 12/20/2012 09:54 AM Aaron Marcuse-Kubitza

README.TXT: Data import: recording the import times in inputs/import.stats.xls: Added step to determine the import date using import_date

6974 12/20/2012 09:52 AM Aaron Marcuse-Kubitza

import_date: Added note that Mac and Linux differ in the order they sort the logs in

6973 12/20/2012 09:50 AM Aaron Marcuse-Kubitza

README.TXT: Data import: recording the import times in inputs/import.stats.xls: Updated pattern for new log filename format

6972 12/20/2012 09:47 AM Aaron Marcuse-Kubitza

README.TXT: Data import: recording the import times in inputs/import.stats.xls: Removed extra ./ before bin/import_times

6971 12/20/2012 09:46 AM Aaron Marcuse-Kubitza

import_date: Added note that the time this outputs is the time the first special input finished importing. The import itself generally starts a few minutes before that, and the exact time is in that import's public schema comment.

6970 12/20/2012 09:41 AM Aaron Marcuse-Kubitza

import_date: Removed duplicate Usage message at top of file, which is repeated in the Usage message provided when the program is run with no arguments

6969 12/20/2012 09:40 AM Aaron Marcuse-Kubitza

Added import_date

6968 12/20/2012 09:38 AM Aaron Marcuse-Kubitza

Added mtime

6967 12/20/2012 09:29 AM Aaron Marcuse-Kubitza

lib/common.Makefile: System: Added $(mtime)

6966 12/20/2012 09:27 AM Aaron Marcuse-Kubitza

lib/common.Makefile: $(date): Factored date format out into $(dateFmt)

6965 12/20/2012 09:25 AM Aaron Marcuse-Kubitza

backups/Makefile: Factored $(isMac) out into lib/common.Makefile

6964 12/20/2012 08:30 AM Aaron Marcuse-Kubitza

README.TXT: Data import: tailing logs: Updated pattern for new log filename format

6963 12/19/2012 02:02 PM Aaron Marcuse-Kubitza

schemas/Makefile: Installation: %/publish: Fixed bug where need quotes around source schema name

6962 12/19/2012 01:57 PM Aaron Marcuse-Kubitza

README.TXT: Data import: Moved deletion previous imports before the import, so that full DB backup can be automated

6961 12/19/2012 01:55 PM Aaron Marcuse-Kubitza

README.TXT: Data import: `make backups/vegbien.$version.backup/test`: Added --exclude-schema=public to leave out the previous (now published) import so it doesn't bloat the backup. Note that public is included in the vegbien.$version.backup for the previous import, named according to its version.

6960 12/19/2012 01:49 PM Aaron Marcuse-Kubitza

import_all: after_import(): Added `make backups/TNRS.backup-remake`

6959 12/19/2012 01:46 PM Aaron Marcuse-Kubitza

README.TXT: Data import: Added step to publish the import to the public schema

6958 12/19/2012 01:42 PM Aaron Marcuse-Kubitza

import_all: after_import(): Added export_analytical_db

6957 12/19/2012 01:36 PM Aaron Marcuse-Kubitza

README.TXT: Data import: bin/export_analytical_db: Removed `env public=$version` because export_analytical_db now uses $version as $public when provided

6956 12/19/2012 01:35 PM Aaron Marcuse-Kubitza

README.TXT: Data import: To remake analytical DB: Removed `env public=...` because $version (which replaces $public) is now set automatically by import_all

6955 12/19/2012 01:32 PM Aaron Marcuse-Kubitza

schemas/Makefile: Installation: py_functions/install: Removed `env public=`, which is not needed since $(psqlAsAdminVegbien) does not use psql_script_vegbien (which uses $public)

6954 12/19/2012 01:28 PM Aaron Marcuse-Kubitza

export_analytical_db: Use vegbien_dest to set the default value for $public

6953 12/19/2012 01:21 PM Aaron Marcuse-Kubitza

README.TXT: Data import: If many inputs have errors: Updated command to `make schemas/$version/uninstall` because the current import's schema is now named $version

6952 12/19/2012 01:15 PM Aaron Marcuse-Kubitza

schemas/Makefile: Installation: $(schemas), $(schemasReversed) (used e.g. by `make schemas/reinstall`): Removed public so that when `make schemas/reinstall` is run before an import, it will not remove any active (published) import which resides in the public schema

6951 12/19/2012 01:10 PM Aaron Marcuse-Kubitza

README.TXT: Schema changes: Reinstall public separately from the other schemas so that it will still be reinstalled when schemas/reinstall excludes the public schema to avoid removing any active (published) import

6950 12/19/2012 01:01 PM Aaron Marcuse-Kubitza

vegbien_dest callers: Removed no longer needed explicit setting $prefix to "", because this is now the default value

6949 12/19/2012 01:00 PM Aaron Marcuse-Kubitza

vegbien_dest: Changed default $prefix to "", so that the majority of callers don't need to manually set $prefix to "" to avoid it defaulting to out_

6948 12/19/2012 12:45 PM Aaron Marcuse-Kubitza

README.TXT: Data import: Use env var $version, which is now set by import_all, instead of manually inserting the version for <version>

6947 12/19/2012 12:40 PM Aaron Marcuse-Kubitza

vegbien_dest: Also export $version

6946 12/19/2012 12:30 PM Aaron Marcuse-Kubitza

import_all: Run the import directly into a new, already-versioned public schema. This removes the need to manually rename the schema after import, and allows the backup commands to use the stored $version shell variable to refer to the last import.

6945 12/19/2012 12:25 PM Aaron Marcuse-Kubitza

schemas/Makefile: %/publish: Added instruction to run `unset version` after the command, to clear the $version shell variable which will be set by import_all

6944 12/19/2012 12:12 PM Aaron Marcuse-Kubitza

README.TXT: Data import: Replaced <import_name> with <version> because the import name is now just the version

6943 12/19/2012 12:10 PM Aaron Marcuse-Kubitza

README.TXT: Data import: Replaced r<revision> with <version> because the version string is now equal to r<revision>

6942 12/19/2012 12:09 PM Aaron Marcuse-Kubitza

README.TXT: Backups: Replaced <date> with <version> because the date is no longer included in the version string

6941 12/19/2012 12:08 PM Aaron Marcuse-Kubitza

README.TXT: Name archived imports without the "public." prefix so that their backups will work with the new `make backups/%.backup/remove` command, which does not add back the prefix

6940 12/19/2012 11:56 AM Aaron Marcuse-Kubitza

backups/Makefile; $(public*): Don't add a "public." prefix to get the name of the public schema

6939 12/19/2012 11:40 AM Aaron Marcuse-Kubitza

backups/Makefile: Removed no longer used $(rmSchema)

6938 12/19/2012 11:39 AM Aaron Marcuse-Kubitza

backups/Makefile: Use \$(rmSchemaCmd) from lib/common.Makefile instead of \$(rmSchema)

6937 12/19/2012 11:20 AM Aaron Marcuse-Kubitza

vegbien_dest: Use $version as $public when $public not provided. When neither is provided, continue to use "public" and also set $version to that.

6936 12/19/2012 11:11 AM Aaron Marcuse-Kubitza

schemas/Makefile: Installation: rotate: Use just the version, without the "public." prefix

6935 12/19/2012 11:04 AM Aaron Marcuse-Kubitza

schemas/Makefile: Installation: `public/install public%/install`: Generalized to /install to allow public schema versions with any name. This requires moving `/install: %.sql` before it to override it.

6934 12/19/2012 11:00 AM Aaron Marcuse-Kubitza

schemas/Makefile: Installation: Merged public/install and public%/install

6933 12/19/2012 10:54 AM Aaron Marcuse-Kubitza

schemas/Makefile: Installation: Moved %/uninstall to beginning of section because it applies to all schemas

6932 12/19/2012 10:52 AM Aaron Marcuse-Kubitza

schemas/Makefile: Installation: public: Generalized public%/publish to %/publish so that public schema versions don't have to start with public_

6931 12/19/2012 10:34 AM Aaron Marcuse-Kubitza

schemas/Makefile: Installation: %/uninstall: Also display schema delete confirmation for schemas whose name is just the version suffix (r<revision #>)

6930 12/19/2012 10:32 AM Aaron Marcuse-Kubitza

schemas/Makefile: Merged public%/uninstall and %/uninstall

6929 12/19/2012 09:49 AM Aaron Marcuse-Kubitza

lib/common.Makefile: Added version target, which prints the current $(version) value

6928 12/19/2012 09:36 AM Aaron Marcuse-Kubitza

schemas/Makefile: Installation: public: public%/uninstall: Fixed bug where need to remove the specified version of the public schema, not public itself. Generalized $(confirmRmPublicSchema) so it could also be used for named versions of the public schema. Inlined $(rmPublicSchema) since it's now only used in one place.

6927 12/19/2012 09:26 AM Aaron Marcuse-Kubitza

lib/common.Makefile: Revisions: $(version): Use just the revision # to avoid cluttering the schema and log file names with long datetime strings

6926 12/19/2012 09:25 AM Aaron Marcuse-Kubitza

schemas/Makefile: public%/install: schema comment: Include current date/time after version

6925 12/19/2012 09:20 AM Aaron Marcuse-Kubitza

lib/common.Makefile: Replaced no longer used $(date) with function to generate human-readable text date (rather than date to put in filename). Removed leading zeros from date and hour. Added timezone.

6924 12/19/2012 09:07 AM Aaron Marcuse-Kubitza

backups/Makefile: Removed no longer used $(dateFmt), $(mtime)

6923 12/19/2012 08:59 AM Aaron Marcuse-Kubitza

backups/Makefile: Removed %.backup/rotate, because this incorrectly causes the current time rather than the version to be used in the backup filename. The version should instead be specified in the backup filename when it's created.

6922 12/19/2012 08:51 AM Aaron Marcuse-Kubitza

schemas/Makefile: Installation: public: Added public%/publish to replace the current public schema with the given version

6921 12/19/2012 08:37 AM Aaron Marcuse-Kubitza

schemas/Makefile: Installation: public: public/uninstall: Added public%/uninstall as a target to allow uninstalling versions of the public schema

6920 12/19/2012 08:30 AM Aaron Marcuse-Kubitza

schemas/Makefile: Installation: public: public%/install: Add a comment on the schema containing the versioned schema name, so that if the schema is later renamed to just public (i.e. "published" as the current version), it will still be possible to tell which version the public schema came from

6919 12/19/2012 08:22 AM Aaron Marcuse-Kubitza

schemas/Makefile: Installation: public: Added public%/install, to install a version of the public schema

6918 12/19/2012 07:59 AM Aaron Marcuse-Kubitza

schemas/Makefile: Removed unused $(os)

6917 12/19/2012 07:58 AM Aaron Marcuse-Kubitza

schemas/Makefile: Removed unused $(SED)

6916 12/19/2012 06:22 AM Aaron Marcuse-Kubitza

Moved schemas-related commands from root Makefile to schemas/Makefile

6915 12/19/2012 06:15 AM Aaron Marcuse-Kubitza

Makefiles: Factored out common vars/functions into lib/common.Makefile

6914 12/19/2012 05:59 AM Aaron Marcuse-Kubitza

root Makefile: $(psqlNoSearchPath): Merged $(psqlAsBien) into it because it's the only place $(psqlAsBien) is used

6913 12/19/2012 05:56 AM Aaron Marcuse-Kubitza

root Makefile: $(psqlAsBien): Use psql_script_vegbien instead of psql_vegbien, which adds $(psqlOpts) itself

6912 12/19/2012 05:50 AM Aaron Marcuse-Kubitza

schemas/Makefile: Include lib/common.Makefile

6911 12/19/2012 05:23 AM Aaron Marcuse-Kubitza

inputs/import.stats.xls: Reformatted so the first by column import and the comparison by row import will fit on the same page when printed on portrait-mode letter paper

6910 12/19/2012 05:10 AM Aaron Marcuse-Kubitza

inputs/import.stats.xls: Changed import type labels to By row/By column so they would fit into one field, leaving the extra field free to contain the revision #

6909 12/19/2012 05:02 AM Aaron Marcuse-Kubitza

lib/common.Makefile: Revisions: Allow $(version) to be overridden in the environment, so that the public schema and all log files share the same, pregenerated version

6908 12/19/2012 04:16 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: Merged provider_view, provider_count, and owner_count into provider_count, using the combining query for Brad's data providers page at <http://bien.nceas.ucsb.edu/bien/people/data-providers/>

6907 12/19/2012 01:23 AM Aaron Marcuse-Kubitza

schemas/vegbien.sql: sync_taxon_trait_to_view(): Changed pkey to index because there can be multiple values of the same taxon's trait from different observations

6906 12/19/2012 01:16 AM Aaron Marcuse-Kubitza

mappings/Makefile: VegCore.csv: Filter out the VegCore tables so they are not matched as terms. This is necessary because some terms have the same name as a table, but the term should be the match rather than the table.

6905 12/19/2012 12:29 AM Aaron Marcuse-Kubitza

sql.py: DbConn.col_info(): raising sql_gen.NoUnderlyingTableException: Fixed bug where also need to catch DoesNotExistException, which is thrown by ::regclass

6904 12/19/2012 12:26 AM Aaron Marcuse-Kubitza

sql.py: DbConn.col_info(): Fixed bug where need to run run_query() recoverably, because this query throws an exception if the column's table does not exist (the information_schema query just returned no rows)

6903 12/19/2012 12:22 AM Aaron Marcuse-Kubitza

sql.py: DbConn.col_info(): Fixed bug where need to use pg_get_expr() on pg_attrdef.adbin instead of shortcut field adsrc, because adsrc does not include schema qualifiers on table names (including strings passed to `nextval('..._seq'::regclass)`)

6902 12/18/2012 11:42 PM Aaron Marcuse-Kubitza

sql.py: DbConn.col_info(): Fixed bug where need to pass through cacheable param to run_query()

6901 12/18/2012 11:41 PM Aaron Marcuse-Kubitza

sql.py: DbConn.col_info(): Fixed bug where need to use .to_str(self) instead of self.esc_value() because self.esc_value() expects a value, not a sql_gen.Literal instance

6900 12/18/2012 11:34 PM Aaron Marcuse-Kubitza

sql.py: DbConn.col_info(): Fixed bug where self needs to be used everywhere that db normally is, because this is a DbConn method rather than a global function

6899 12/18/2012 11:31 PM Aaron Marcuse-Kubitza

sql.py: DbConn.col_info(): For PostgreSQL, use pg_catalog tables directly instead of their views in information_schema. This allows using ::regclass to look up the table in the search_path, and fixes a bug in imports with an explicit public schema where column types were looked up in public instead of public.<version>. Also don't wrap default using sql_gen.as_Code() when it's None (indicating no default value, aka default=NULL), because this value is interpreted specially by sql_gen.TypedCol.

6898 12/18/2012 11:06 PM Aaron Marcuse-Kubitza

inputs/Makefile: Input data: $(rsyncSrcs): Also include log files other than install.log.sql

6897 12/18/2012 09:41 PM Aaron Marcuse-Kubitza

import_all: Run all imports (not just the main datasources' import) with $import_source turned off, so that the Source tables will not be imported a second time when the datasource's main tables are imported. Note that it's not necessary to wait for asynchronous commands after the jobs for the main import are started (so that $import_source is not unset until after they are started), because with_all does not return until all jobs are started and have noted the $import_source setting in effect in the shell environment.

6896 12/18/2012 09:32 PM Aaron Marcuse-Kubitza

import_all: Source tables import: Fixed bug where need to use $all option to with_all to also include special datasources starting with "."

6895 12/18/2012 09:23 PM Aaron Marcuse-Kubitza

make_analytical_db: Also create taxon_trait materialized view

6894 12/17/2012 08:17 PM Aaron Marcuse-Kubitza

inputs/*/*/map.csv: Reverted special OMIT mappings for input columns that have the same name as a VegCore table and have not yet been mapped to a VegCore term

6893 12/17/2012 08:06 PM Aaron Marcuse-Kubitza

mappings/Makefile: VegCore.csv: Filter out the VegCore tables so they are not matched as terms. This is necessary because some terms have the same name as a table, but the term should be the match rather than the table.

6892 12/17/2012 08:04 PM Aaron Marcuse-Kubitza

mappings/VegCore.csv: Changed line endings to \r\n to match the output of filter_out_ci

6891 12/17/2012 05:51 PM Aaron Marcuse-Kubitza

inputs/CTFS/TaxonOccurrence/map.csv: Mapped SpeciesAuthority

6890 12/17/2012 04:59 PM Aaron Marcuse-Kubitza

backups/Makefile: Synchronization: $(remote): Fixed bug where need trailing / at end of path