Project

General

Profile

Statistics
| Revision:

# Date Author Comment
10149 07/03/2013 04:39 PM Aaron Marcuse-Kubitza

schemas/util.sql: set_col_names(): don't perform rename if the name is not changing, to avoid cluttering the debug output with unnecessary queries

10148 07/03/2013 04:21 PM Aaron Marcuse-Kubitza

lib/runscripts/table.run: use new util.set_col_names_with_metadata() instead of util.set_col_names() so that metadata values (beginning with : ) are automatically mapped to constant columns rather than needing to add a mk_const_col() call to postprocess.sql for each of them. there are a lot of metadata value entries, especially in the Source/ tables for each datasource, so this will save time in translating the datasources to new-style import. note that this requires disabling the map_filter_insert trigger on the map table to prevent it from filtering out the metadata entries before util.set_col_names_with_metadata() can use them.

10147 07/03/2013 03:55 PM Aaron Marcuse-Kubitza

bugfix: schemas/util.sql: set_col_names_with_metadata(): need `util.` before mk_const_col(). "to", "from" need to be referenced from row_. substring() needs to start from 2 rather than 1 because PostgreSQL string indexes are 1-based.

10146 07/03/2013 03:05 PM Aaron Marcuse-Kubitza

schemas/util.sql: try_create(), create_if_not_exists(): use eval() so the executed statement will be echoed for debugging

10145 07/03/2013 02:58 PM Aaron Marcuse-Kubitza

schemas/util.sql: added set_col_names_with_metadata()

10144 07/02/2013 05:42 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/sync.sh: upload(): paths: don't dereference the path itself if it's a symlink; instead canonicalize just its parent dir. this allows syncing a specific file which is a symlink, rather than syncing the symlink's target.

10143 07/02/2013 05:40 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: added canon_dir_rel_path(), which canonicalizes just the parent dir if the path is a symlink, to leave the symlink itself untouched

10142 07/02/2013 05:08 PM Aaron Marcuse-Kubitza

planning/workflow/validation/: archived BIEN2 validations documents which have been superseded by planning/goals/BIEN3_derived_data_products.docx, to avoid confusion

10141 07/02/2013 04:45 PM Aaron Marcuse-Kubitza

planning/workflow/bien3_architecture.pptx: updated with clarifications made in today's conference call

10140 07/02/2013 02:31 PM Aaron Marcuse-Kubitza

bugfix: bin/map: in_is_db: inline metadata value columns (used by new-style import) so that they can be compared by value in XML simplifying functions (lib/xml_func.py)

10139 07/02/2013 02:29 PM Aaron Marcuse-Kubitza

lib/sql.py: added col_default_value(), col_is_constant(), which interface with corresponding util-schema functions

10138 07/02/2013 02:28 PM Aaron Marcuse-Kubitza

lib/sql_gen.py: added col2col_ref() for interfacing with for SQL functions that take a util.col_ref

10137 07/02/2013 12:57 PM Aaron Marcuse-Kubitza

schemas/util.sql: added is_constant(col_ref), for checking if a column has been marked "constant"

10136 07/02/2013 12:54 PM Aaron Marcuse-Kubitza

schemas/util.sql: added col_comment()

10135 07/02/2013 12:53 PM Aaron Marcuse-Kubitza

schemas/util.sql: mk_const_col(): add column comment "constant" to mark column as inlinable (needed by some mappings to have a literal value to compare)

10134 07/02/2013 12:03 PM Aaron Marcuse-Kubitza

schemas/util.sql: added col_default_value(), which evaluates the col_default_sql() expression

10133 07/02/2013 11:51 AM Aaron Marcuse-Kubitza

schemas/util.sql: added eval_expr_passthru() (passes NULL SQL through)

10132 07/02/2013 11:45 AM Aaron Marcuse-Kubitza

bugfix: schemas/util.sql: eval_expr(): need to pass ret_type_null to eval2val()

10131 07/02/2013 11:42 AM Aaron Marcuse-Kubitza

schemas/util.sql: added eval_expr() (does not require `SELECT ` before expr)

10130 07/02/2013 11:33 AM Aaron Marcuse-Kubitza

schemas/util.sql: added col_default_sql()

10129 07/02/2013 11:26 AM Aaron Marcuse-Kubitza

schemas/util.sql: eval(text, anyelement): added default polymorphic type text (can't be unknown because this would cause a "could not determine polymorphic type because input has type "unknown"" error). renamed to eval2val() to avoid overloading conflicts with eval(text) when no polymorphic type param is specified.

10128 07/02/2013 11:15 AM Aaron Marcuse-Kubitza

schemas/util.sql: added value-returning eval()

10127 07/02/2013 11:02 AM Aaron Marcuse-Kubitza

bugfix: lib/common.Makefile: $(asAdmin): need to use _postgres instead on Mac for OS X 10.8 Mountain Lion

10126 07/02/2013 11:01 AM Aaron Marcuse-Kubitza

bugfix: *Makefile: $(asAdmin) invocations of Postgres commands: need to set DB user to postgres so that it won't default to the system user _postgres

10125 07/02/2013 10:57 AM Aaron Marcuse-Kubitza

*Makefile: removed $(psqlOpts), $(psqlAsAdmin), which are now set by lib/common.Makefile

10124 07/02/2013 10:57 AM Aaron Marcuse-Kubitza

lib/common.Makefile: added $(psqlOpts), $(psqlAsAdmin)

10123 07/02/2013 10:54 AM Aaron Marcuse-Kubitza

bugfix: schemas/pg_hba.Mac.conf: use new postgres ident map instead of changing user to _postgres, because the DB user is still named postgres

10122 07/02/2013 10:53 AM Aaron Marcuse-Kubitza

schemas/pg_ident.Mac.conf: added postgres map mapping the _postgres system user to the postgres DB user for ident authentication

10121 07/02/2013 10:45 AM Aaron Marcuse-Kubitza

/Makefile: $(postgresReload-Darwin): also install pg_ident.Mac.conf

10120 07/02/2013 10:44 AM Aaron Marcuse-Kubitza

placed pg_ident.conf under version control as schemas/pg_ident.Mac.conf

10119 07/02/2013 10:29 AM Aaron Marcuse-Kubitza

*Makefile: removed $(asAdmin), which is now set by lib/common.Makefile

10118 07/02/2013 10:28 AM Aaron Marcuse-Kubitza

lib/common.Makefile: added $(asAdmin)

10117 07/02/2013 10:26 AM Aaron Marcuse-Kubitza

bugfix: schemas/pg_hba.Mac.conf: changed postgres to _postgres for OS X 10.8 Mountain Lion

10116 07/02/2013 09:48 AM Aaron Marcuse-Kubitza

schemas/util.sql: added raise_undefined_column() for use in translating other exceptions to undefined_column

10115 07/02/2013 03:50 AM Aaron Marcuse-Kubitza

bin/map: map_table(): Resolve prefixes: combined db_xml.ColRef() constructor call with creation of args (as tuple) for clarity

10114 07/02/2013 03:35 AM Aaron Marcuse-Kubitza

bin/map: update_in_label(): use in_schema instead of the map spreadsheet column name when available, to allow using one spreadsheet for all datasources (which would not have a datasource-specific spreadsheet column name)

10113 07/02/2013 02:59 AM Aaron Marcuse-Kubitza

schemas/util.sql: added mk_source_col(), which uses the schema name instead of the map spreadsheet header to get the datasource name

10112 07/02/2013 02:44 AM Aaron Marcuse-Kubitza

schemas/util.sql: added table_schema()

10111 07/02/2013 01:15 AM Aaron Marcuse-Kubitza

added planning/goals/iPlant_BIEN_Proposal_Final.pdf with Mark's e-mail notes in iPlant_BIEN_Proposal_Final.pdf.notes.txt

10110 07/01/2013 11:52 PM Aaron Marcuse-Kubitza

bugfix: schemas/util.sql: mk_map_table(): add trigger to remove metadata value entries from the map spreadsheet as it's being imported, since these do not correspond to actual columns in the staging table and will produce an error when trying to rename nonexistent columns to them. note that these entries will still need to be included properly in VegBIEN.csv, which will require changes to the VegBIEN.csv make target.

10109 06/28/2013 04:54 PM Aaron Marcuse-Kubitza

empty inputs/*/import_order.txt: added subdirs in the order they are used by inputs/input.Makefile, by running make on the inputs to auto-populate import_order.txt. import_order.txt is needed by the runscripts to run the right set of subdirs in the right order.

10108 06/28/2013 04:48 PM Aaron Marcuse-Kubitza

added inputs/.TNRS/grants.sql, with statements to provide SELECT access to bien_read. these statements must be in grants.sql to avoid them being filtered out by pg_dump_limit.

10107 06/28/2013 04:47 PM Aaron Marcuse-Kubitza

inputs/input.Makefile: added support for separate grants.sql file, which may contain GRANT statements that would normally be filtered out by pg_dump_limit

10106 06/28/2013 04:44 PM Aaron Marcuse-Kubitza

inputs/input.Makefile: sql/install: added $debug option to run the *.sql import verbosely, to display which statements are being run. this should only be used for SQL files that use COPY FROM to import data, to avoid echoing pages of insert statements.

10105 06/28/2013 01:53 PM Aaron Marcuse-Kubitza

inputs/input.Makefile: keep $(sortFile) up-to-date: use sort_file_updated=1 flag to indicate that import_order.txt has already been checked, so that recursive invocations of make don't need to recheck it. also use this flag instead of an explicit $(MAKECMDGOALS) list to prevent the $(sortFile) check from being infinite-recursively reinvoked when input.Makefile is read as part of the $(sortFile) check itself.

10104 06/28/2013 01:38 PM Aaron Marcuse-Kubitza

inputs/input.Makefile: keep import_order.txt up-to-date by running `make $(sortFile)` each time make is run. this ensures that new datasources always have import_order.txt populated when make is first run. eventually, $(tables) can be always set to $(allTables) so that this auto-updating can also be used to ensure that new subdirs added by the user always make it into import_order.txt (so that they will be included in the subdirs that get remade, etc.). import_order.txt is primarily for specifying the order of the subdirs, but some datasources also use it to filter out subdirs, so it can't yet be always updated to include the full list of subdirs. however, the filter-out usage should no longer be necessary after the switch to new-style import.

10103 06/28/2013 12:58 PM Aaron Marcuse-Kubitza

inputs/input.Makefile: added $(filter_make), used to filter the output of embedded $(shell make ...) invocations

10102 06/28/2013 11:39 AM Aaron Marcuse-Kubitza

inputs/input.Makefile: $(sortFile): use $(filter-out)->then instead of $(filter)->else for clarity

10101 06/28/2013 11:21 AM Aaron Marcuse-Kubitza

inputs/input.Makefile: added $(sortFile) (import_order.txt) target which adds any missing tables to import_order.txt

10100 06/28/2013 11:03 AM Aaron Marcuse-Kubitza

inputs/input.Makefile: added list_tables to print $(tables) for use in populating import_order.txt

10099 06/28/2013 02:50 AM Aaron Marcuse-Kubitza

web/links/index.htm: updated to Firefox bookmarks. grouped version control systems into new version control folder.

10098 06/27/2013 09:54 PM Aaron Marcuse-Kubitza

inputs/.NCBI/: added new-style import runscripts, which renamed the staging table columns to VegCore

10097 06/27/2013 04:48 PM Aaron Marcuse-Kubitza

bugfix: lib/runscripts/datasrc_dir.run, subdir.run: need to remove leading . from dir name to get installed schema name, using new dir2schema()

10096 06/27/2013 04:47 PM Aaron Marcuse-Kubitza

lib/runscripts/datasrc_dir.run, subdir.run: use new lib/sh/datasrc.sh, which contains code in common to both datasrc-related dir runscripts

10095 06/27/2013 04:46 PM Aaron Marcuse-Kubitza

added lib/sh/datasrc.sh

10094 06/27/2013 03:47 PM Aaron Marcuse-Kubitza

inputs/.TNRS/schema.sql: AcceptedTaxon: removed Annotations entry because the accepted name only contains name elements, not additional text (vegpath.org/cf_aff)

10093 06/27/2013 01:02 PM Aaron Marcuse-Kubitza

bugfix: /README.TXT: Maintenance: syncing ~/bien to ~/Dropbox/svn: added overwrite=1 so that perms transfer from the authoritative ~/bien regardless of relative mtimes

10092 06/27/2013 12:45 PM Aaron Marcuse-Kubitza

removed no longer used lib/import.sh. use lib/runscripts/table.run instead.

10091 06/27/2013 12:28 PM Aaron Marcuse-Kubitza

added inputs/*/*/header.csv for CSV inputs, which are now generated by inputs/input.Makefile %/install

10090 06/27/2013 12:23 PM Aaron Marcuse-Kubitza

added inputs/FIA/*/{VegBIEN.csv,test.xml.ref}, which are now generated by the mapping process for the joined-together tables (even though they are not used by the import, because only occurrence_all is imported)

10089 06/27/2013 12:20 PM Aaron Marcuse-Kubitza

added inputs/GBIF/_archive/

10088 06/27/2013 12:18 PM Aaron Marcuse-Kubitza

removed inputs/GBIF/Specimen/, which has been replaced by the refresh in raw_occurrence_record_plants/

10087 06/27/2013 12:17 PM Aaron Marcuse-Kubitza

added inputs/GBIF/map.csv, used to regenerate inputs/GBIF/raw_occurrence_record_plants/map.csv when raw_occurrence_record_plants is resubset

10086 06/27/2013 12:12 PM Aaron Marcuse-Kubitza

inputs/FIA/*/postprocess.sql: removed svn:executable attribute using `svn pdel svn:executable ...` now that these are not shell scripts

10085 06/27/2013 12:11 PM Aaron Marcuse-Kubitza

removed no longer needed inputs/FIA/import. use inputs/FIA/run instead.

10084 06/27/2013 12:10 PM Aaron Marcuse-Kubitza

inputs/FIA/*/import: changed to postprocess.sql for use by the runscripts

10083 06/27/2013 04:27 AM Aaron Marcuse-Kubitza

added inputs/FIA/run

10082 06/27/2013 04:26 AM Aaron Marcuse-Kubitza

added inputs/FIA/*/run. these do not yet use the postprocessing operations in */import.

10081 06/27/2013 04:24 AM Aaron Marcuse-Kubitza

added inputs/FIA/table.run (for use by table subdirs) and helper Makefile

10080 06/27/2013 04:17 AM Aaron Marcuse-Kubitza

added lib/runscripts/view.run, for use with table subdirs for views, such as inputs/FIA/occurrence_all/

10079 06/27/2013 02:14 AM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: added Reload analytical database checkmark for every Rebuild core database checkmark, because these are always done together as part of the import process

10078 06/27/2013 01:41 AM Aaron Marcuse-Kubitza

bugfix: inputs/FIA/occurrence_all/import: don't re-prepend * to terms because this is a view, and the underlying columns have already been mapped

10077 06/27/2013 01:40 AM Aaron Marcuse-Kubitza

bin/src_map: support custom (or no) new_term_prefix. no new_term_prefix is useful for views whose columns have already been renamed in the underlying tables and should not have * re-prepended.

10076 06/27/2013 01:03 AM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: moved longer-term goals to new August column, leaving near-term goals in July

10075 06/27/2013 01:00 AM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: erased cells where a task was planned but not worked on, so that all shaded cells in the past have check marks to indicate completion of a portion of the task, and empty shaded cells in the future indicate work left to do

10074 06/27/2013 12:50 AM Aaron Marcuse-Kubitza

planning/timeline/timeline.2013.xls: updated for current progress. renamed "Rerun species range models" to "Prepare to rerun species range models" because the range modeling itself is not part of the BIEN DB development. added a column for July with the tasks that are not yet complete.

10073 06/26/2013 06:57 PM Aaron Marcuse-Kubitza

bugfix: inputs/FIA/REF_SPECIES/import: PLANT_SYMBOL_TYPE: prepended * since it's a datasource column, and needs to match up with *PLANT_SYMBOL_TYPE in other table for joins

10072 06/26/2013 06:57 PM Aaron Marcuse-Kubitza

bugfix: inputs/FIA/REF_SPECIES/import: PLANT_SYMBOL_TYPE: prepended * since it's a datasource column, and needs to match up with *PLANT_SYMBOL_TYPE in other table for joins

10071 06/26/2013 06:23 PM Aaron Marcuse-Kubitza

schemas/util.sql: try_create(): also ignore wrong_object_type exceptions thrown when trying to alter a view's columns

10070 06/26/2013 03:36 PM Aaron Marcuse-Kubitza

added inputs/FIA/_src/run, which runs ./download

10069 06/26/2013 03:00 PM Aaron Marcuse-Kubitza

lib/sh/make.sh: make(): run sys_cmd_path at a higher log_level since the make() steps should not be displayed by default

10068 06/26/2013 02:58 PM Aaron Marcuse-Kubitza

/README.TXT: to synchronize vegbiendev, jupiter, and your local machine: added step to update mtimes/perms on ~/Dropbox/svn/ so that copying files back to ~/bien does not overwrite the permissions from what is on vegbiendev

10067 06/26/2013 02:44 PM Aaron Marcuse-Kubitza

inputs/: don't upload test*.xml to jupiter on vegbiendev, because these files are also generated by the full database import but should only be backed up from one source machine, starscream (the Mac)

10066 06/26/2013 02:26 PM Aaron Marcuse-Kubitza

bin/make: moved $make_filter_active test to lib/sh/make.sh make() so that it's also used when make() is run directly (e.g. in a runscript) rather than via the bin/make wrapper in the PATH

10065 06/26/2013 02:22 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/make.sh: make(): need to match absolute `make` paths such as /usr/bin/make

10064 06/26/2013 02:19 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: added self_name alias and use it in self/self_sys

10063 06/26/2013 02:18 PM Aaron Marcuse-Kubitza

lib/sh/util.sh: added sys_cmd_path() and use it in cmd2sys

10062 06/26/2013 01:05 PM Aaron Marcuse-Kubitza

bugfix: bin/make: use separate $make_filter_active flag instead of $is_outermost for avoiding duplicate output filtering, so that an outer runscript, which sets $is_outermost but does not activate the make filter, will not prevent the make filter from being activated when make is invoked

10061 06/26/2013 01:00 PM Aaron Marcuse-Kubitza

bugfix: bin/make: need to use sys_cmd instead of command so that the system make command is invoked instead of the wrapper (which would cause infinite mutual recursion for the ~/bien working copy, although not for the ~/Dropbox/svn working copy because nonrecursive=1 was able to remove the single recursion)

10060 06/26/2013 12:19 PM Aaron Marcuse-Kubitza

bin/make: use .rel to do relative includes

10059 06/26/2013 12:19 PM Aaron Marcuse-Kubitza

bugfix: lib/sh/util.sh: .rel(): first use realpath() on BASH_SOURCE1 in case it's a symlink (as it is for bin/make)

10058 06/26/2013 12:00 PM Aaron Marcuse-Kubitza

inputs/FIA/_src/Makefile: Extraction: $(zips): use $(allZips) containing a zip for each state so that states that have not yet been downloaded and extracted (or had an empty dir created for them) will be downloaded. previously, the extract target only expanded existing zips but did not download new zips unless no zips had yet been downloaded. (this had been necessary because some states do not have a download, and the download of them would be continuously retried every time the Makefile was run.)

10057 06/26/2013 11:51 AM Aaron Marcuse-Kubitza

bugfix: inputs/FIA/_src/Makefile: `%: %.zip`: if unzip fails because the download does not exist, create an empty dir for the state instead of aborting make

10056 06/26/2013 11:33 AM Aaron Marcuse-Kubitza

inputs/FIA/_src/Makefile: use curl instead of wget because that is also available on Mac

10055 06/26/2013 11:32 AM Aaron Marcuse-Kubitza

bugfix: lib/sh/web.sh: curl(): use --fail so that curl returns a nonzero exit status on error (e.g. file not found) instead of appearing to exit successfully but outputting an error HTML document instead of the file

10054 06/26/2013 11:05 AM Aaron Marcuse-Kubitza

inputs/FIA/SUBPLOT/map.csv, import: prepended * to all FIA terms to clearly distinguish them from the VegCore terms. this is the standard convention for all datasources, to indicate which terms have not yet been mapped, but was not yet implemented at the beginning of new-style import (the FIA refresh was the first new-style datasource)....

10053 06/26/2013 08:59 AM Aaron Marcuse-Kubitza

inputs/FIA/import_order.txt: added remaining src tables, whose runscripts will be invoked in the order listed by lib/runscripts/datasrc_dir.run

10052 06/26/2013 08:58 AM Aaron Marcuse-Kubitza

added inputs/FIA/*/_no_import to src tables that are joined together in occurrence_all and should not also be imported separately once they are in import_order.txt

10051 06/26/2013 07:55 AM Aaron Marcuse-Kubitza

inputs/GBIF/run: inherit from lib/runscripts/datasrc_dir.run, which uses import_order.txt to forward calls to the subdirs

10050 06/26/2013 07:54 AM Aaron Marcuse-Kubitza

added blank runscripts inputs/GBIF/Source/run, Specimen/run because they are in import_order.txt (used by lib/runscripts/datasrc_dir.run)