Activity
From 06/09/2013 to 07/08/2013
07/08/2013
07/06/2013
- 07:29 PM Revision 10189: inputs/.herbaria/: switched to new-style import, which renamed the columns to the VegCore names. this is done using the commands at wiki.vegpath.org/2013-06-27_conference_call#To-do-for-Aaron > "run the following for each datasource".
- 07:21 PM Revision 10188: lib/sql_io.py: cleanup_table(): don't run the slow ALTER TABLE statement again if the table has already been cleaned up. documented that it is idempotent (and actually was before this change as well).
- 07:19 PM Revision 10187: lib/sql_io.py: added table_nulls_mapped__set(), "__get() wrappers around the corresponding util schema functions
- 07:18 PM Revision 10186: lib/sql_gen.py: added table2regclass_text()
- 07:07 PM Revision 10185: schemas/util.sql: added table_nulls_mapped__get(), which gets whether a table's NULL-equivalent strings have been replaced with NULL
- 07:06 PM Revision 10184: schemas/util.sql: added table_flag__get(), which gets whether a status flag is set by the presence of a table constraint
- 06:56 PM Revision 10183: schemas/util.sql: added table_nulls_mapped__set(), which sets that a table's NULL-equivalent strings have been replaced with NULL
- 06:54 PM Revision 10182: schemas/util.sql: added table_flag__set(), which stores a status flag by the presence of a table constraint
- 06:52 PM Revision 10181: schemas/util.sql: create_if_not_exists(): also ignore duplicate_object exceptions, thrown when trying to add a duplicate constraint
- 06:00 PM Revision 10180: inputs/input.Makefile: %/postprocess: removed no longer used invocation of $*/import (precursor to the runscripts used in FIA)
- 05:39 PM Revision 10179: inputs/*/: added table.run for use by the table subdirs in new-style import. datasources without table subdirs do not need this.
- 05:35 PM Revision 10178: inputs/*/: added top-level Makefile which includes inputs/input.Makefile, so that make can be run directly on the datasrc dir without needing to specify `--makefile=../input.Makefile` (see input.Makefile $(selfMake))
- 05:17 PM Revision 10177: inputs/*/: added top-level Makefile which includes inputs/input.Makefile, so that make can be run directly on the datasrc dir without needing to specify `--makefile=../input.Makefile` (see input.Makefile $(selfMake))
- 05:05 PM Revision 10176: added inputs/test_taxonomic_names/Taxon/header.csv
- 04:02 PM Revision 10175: web/links/index.htm: updated to Firefox bookmarks. removed dead favicons. PostgreSQL: added bookmarks about triggers.
- 03:55 PM Revision 10174: bugfix: inputs/input.Makefile: %/VegBIEN.csv: for new-style datasources, use a symlink to mappings/VegCore-VegBIEN.csv directly instead of prefiltering VegCore-VegBIEN.csv to include only the columns in map.csv. prefiltering used to be performed as part of mapping the map.csv VegCore output terms to VegBIEN using bin/join, but is no longer needed because the staging table columns are now VegCore terms. instead, the full VegCore-VegBIEN.csv *is* needed so that derived columns added in stage I or II validations are detected by bin/map (rather than just the original source columns in map.csv).
- 03:37 PM Revision 10173: mappings/VegCore-VegBIEN.csv: cultivated, oldGrowth: use just cultivated if it's provided, rather than /_alt-ing it back with oldGrowth (which it was generated from)
- 03:30 PM Revision 10172: bugfix: mappings/VegCore-VegBIEN.csv: fixed priority of cultivated and oldGrowth so cultivated is used first if it's available
- 02:41 PM Revision 10171: bugfix: lib/runscripts/table.run: need to run remake_VegBIEN_mappings after mk_derived rather than before so the derived cols will be included in the automated test result
- 02:26 PM Revision 10170: bugfix: inputs/*/Source/: use installed staging table (with blank-line data.csv) in order to also work with new-style import. this also fixes a benign diff between the by-row and by-col test outputs, where row-based import would not import the Source/ entries because there was not at least one row in the input. note that in order to ensure that all datasources are properly run, you need to check `svn st|sort` against the datasource schema names to see if any are missing.
- 02:22 PM Revision 10169: inputs/*/logs: updated svn:ignore
- 02:22 PM Revision 10168: inputs/*/*/logs: updated svn:ignore
- 01:45 PM Revision 10167: bugfix: inputs/input.Makefile: SVN: add: don't add subdirs for datasources marked _no_import (e.g. datasources which only have an inputs/ dir to be listed in VegPath)
- 11:29 AM Revision 10166: bugfix: inputs/*/Source/data.csv for new-style datasources: need to include a blank row (plus a blank header) so that the metadata values are imported at least once instead of zero times, now that there is an installed staging table that will be iterated over. the blank row did not used to be necessary, because db_xml.put_table() has a special case for metadata-only tables with no installed table, which avoids iterating over the table's rows.
07/03/2013
- 10:48 PM Revision 10165: lib/sql_io.py: put_table() (column-based import): complexity note: clarified that INSERT RETURNING throws an error *on duplicate* instead of returning the existing row. added blank line after ¶ for readability.
- 10:44 PM Revision 10164: lib/sql_io.py: put_table() (column-based import): warning about triggers populating unique constraint-covered columns: corrected limitation to include only *the* unique constraint used to do the DISTINCT ON, since other unique constraints are not affected by column-based import. note that the primary key will normally not be the DISTINCT ON constraint, so trigger-populated natural keys are supported *unless* the input table contains duplicate rows for some generated keys.
- 10:20 PM Revision 10163: inputs/*/Source/ for new-style datasources: use an actual staging table instead of a metadata-only table, so that metadata values can be stored in the staging table instead of the map.csv (as will be required by new-style import)
- 08:21 PM Revision 10162: inputs/input.Makefile: SVN: $(svnFilesGlob): added data.csv, used to store versioned data (such as the empty data.csv used by Source/ tables which have their metadata in the map table instead)
- 07:45 PM Revision 10161: schemas/util.sql: type_qual(), type_qual_name(): added comments to distinguish these similarly-named functions, one of which gets a type qualifier and the other of which gets a qualified name (not the name of a type qualifier, which one might otherwise assume)
- 07:39 PM Revision 10160: schemas/util.sql: typeof(): support expressions that are not relative to a table (which do not have a table_ param). note that this requires removing the STRICT qualifier, so that NULL expressions will now produce an error instead of passing through as NULL.
- 07:10 PM Revision 10159: schemas/VegCore/VegCore.ERD.mwb: relationships legend: removed inheritance of base_class from record, so that the IS-A label would not confusingly appear to apply to the record connector stub instead of to the solid line between base_class and derived_class
- 06:51 PM Revision 10158: bugfix: schemas/util.sql: col_names(): need to exclude dropped columns (which remain included in the pg_attribute table until the next tuple rewrite), by filtering on `NOT attisdropped`. lib/sql.py table_col_names() is not affected by this because it is able to access the column names from the DB driver directly, after performing `SELECT * FROM table LIMIT 0`.
- 06:38 PM Revision 10157: schemas/util.sql: set_col_names_with_metadata(): don't delete the metadata entries from the map table, because they are now added *before* the renames take place, so that the renames can simply be performed on the constant columns themselves. this does, however, require that the metadata entries are always listed *last* in the map.csv (which is currently the case).
- 05:56 PM Revision 10156: lib/runscripts/table.run: map_table(): store the map table in the datasource schema, so that it can easily be referred to when using the staging tables. this also allows it to be found more easily when debugging its contents.
- 05:26 PM Revision 10155: lib/sh/db.sh: psql(): hide the verbose CONTEXT information that is output with each NOTICE by setting the VERBOSITY psql var to terse (postgresql.1045698.n5.nabble.com/Quiet-quot-CONTEXT-quot-td1906036.html#a1906037)
- 05:15 PM Revision 10154: *{.sh,run}: use new log-() instead of log+() with a negative #
- 05:14 PM Revision 10153: lib/sh/util.sh: added log-() because it's non-obvious that you would otherwise have to invoke log+() with a negative #
- 05:00 PM Revision 10152: schemas/util.sql: reset_map_table(): drop the table and recreate it instead of just creating it if it doesn't exist, so that any change to the util.map table is propagated to persistent map tables whenever they are reloaded from the map.csv
- 05:00 PM Revision 10151: lib/runscripts/table.run: map_table(): create the map table as a persistent table in the temp schema, so that its contents can be viewed for debugging
- 04:50 PM Revision 10150: schemas/util.sql: added drop_table()
- 04:39 PM Revision 10149: schemas/util.sql: set_col_names(): don't perform rename if the name is not changing, to avoid cluttering the debug output with unnecessary queries
- 04:21 PM Revision 10148: lib/runscripts/table.run: use new util.set_col_names_with_metadata() instead of util.set_col_names() so that metadata values (beginning with : ) are automatically mapped to constant columns rather than needing to add a mk_const_col() call to postprocess.sql for each of them. there are a lot of metadata value entries, especially in the Source/ tables for each datasource, so this will save time in translating the datasources to new-style import. note that this requires disabling the map_filter_insert trigger on the map table to prevent it from filtering out the metadata entries before util.set_col_names_with_metadata() can use them.
- 03:55 PM Revision 10147: bugfix: schemas/util.sql: set_col_names_with_metadata(): need `util.` before mk_const_col(). "to", "from" need to be referenced from row_. substring() needs to start from 2 rather than 1 because PostgreSQL string indexes are 1-based.
- 03:05 PM Revision 10146: schemas/util.sql: try_create(), create_if_not_exists(): use eval() so the executed statement will be echoed for debugging
- 02:58 PM Revision 10145: schemas/util.sql: added set_col_names_with_metadata()
07/02/2013
- 05:42 PM Revision 10144: bugfix: lib/sh/sync.sh: upload(): paths: don't dereference the path itself if it's a symlink; instead canonicalize just its parent dir. this allows syncing a specific file which is a symlink, rather than syncing the symlink's target.
- 05:40 PM Revision 10143: lib/sh/util.sh: added canon_dir_rel_path(), which canonicalizes just the parent dir if the path is a symlink, to leave the symlink itself untouched
- 05:08 PM Revision 10142: planning/workflow/validation/: archived BIEN2 validations documents which have been superseded by planning/goals/BIEN3_derived_data_products.docx, to avoid confusion
- 04:45 PM Revision 10141: planning/workflow/bien3_architecture.pptx: updated with clarifications made in today's conference call
- 02:31 PM Revision 10140: bugfix: bin/map: in_is_db: inline metadata value columns (used by new-style import) so that they can be compared by value in XML simplifying functions (lib/xml_func.py)
- 02:29 PM Revision 10139: lib/sql.py: added col_default_value(), col_is_constant(), which interface with corresponding util-schema functions
- 02:28 PM Revision 10138: lib/sql_gen.py: added col2col_ref() for interfacing with for SQL functions that take a util.col_ref
- 12:57 PM Revision 10137: schemas/util.sql: added is_constant(col_ref), for checking if a column has been marked "constant"
- 12:54 PM Revision 10136: schemas/util.sql: added col_comment()
- 12:53 PM Revision 10135: schemas/util.sql: mk_const_col(): add column comment "constant" to mark column as inlinable (needed by some mappings to have a literal value to compare)
- 12:03 PM Revision 10134: schemas/util.sql: added col_default_value(), which evaluates the col_default_sql() expression
- 11:51 AM Revision 10133: schemas/util.sql: added eval_expr_passthru() (passes NULL SQL through)
- 11:45 AM Revision 10132: bugfix: schemas/util.sql: eval_expr(): need to pass ret_type_null to eval2val()
- 11:42 AM Revision 10131: schemas/util.sql: added eval_expr() (does not require `SELECT ` before expr)
- 11:33 AM Revision 10130: schemas/util.sql: added col_default_sql()
- 11:26 AM Revision 10129: schemas/util.sql: eval(text, anyelement): added default polymorphic type text (can't be unknown because this would cause a "could not determine polymorphic type because input has type "unknown"" error). renamed to eval2val() to avoid overloading conflicts with eval(text) when no polymorphic type param is specified.
- 11:15 AM Revision 10128: schemas/util.sql: added value-returning eval()
- 11:02 AM Revision 10127: bugfix: lib/common.Makefile: $(asAdmin): need to use _postgres instead on Mac for OS X 10.8 Mountain Lion
- 11:01 AM Revision 10126: bugfix: *Makefile: $(asAdmin) invocations of Postgres commands: need to set DB user to postgres so that it won't default to the system user _postgres
- 10:57 AM Revision 10125: *Makefile: removed $(psqlOpts), $(psqlAsAdmin), which are now set by lib/common.Makefile
- 10:57 AM Revision 10124: lib/common.Makefile: added $(psqlOpts), $(psqlAsAdmin)
- 10:54 AM Revision 10123: bugfix: schemas/pg_hba.Mac.conf: use new postgres ident map instead of changing user to _postgres, because the DB user is still named postgres
- 10:53 AM Revision 10122: schemas/pg_ident.Mac.conf: added postgres map mapping the _postgres system user to the postgres DB user for ident authentication
- 10:45 AM Revision 10121: /Makefile: $(postgresReload-Darwin): also install pg_ident.Mac.conf
- 10:44 AM Revision 10120: placed pg_ident.conf under version control as schemas/pg_ident.Mac.conf
- 10:29 AM Revision 10119: *Makefile: removed $(asAdmin), which is now set by lib/common.Makefile
- 10:28 AM Revision 10118: lib/common.Makefile: added $(asAdmin)
- 10:26 AM Revision 10117: bugfix: schemas/pg_hba.Mac.conf: changed postgres to _postgres for OS X 10.8 Mountain Lion
- 09:48 AM Revision 10116: schemas/util.sql: added raise_undefined_column() for use in translating other exceptions to undefined_column
- 03:50 AM Revision 10115: bin/map: map_table(): Resolve prefixes: combined db_xml.ColRef() constructor call with creation of args (as tuple) for clarity
- 03:35 AM Revision 10114: bin/map: update_in_label(): use in_schema instead of the map spreadsheet column name when available, to allow using one spreadsheet for all datasources (which would not have a datasource-specific spreadsheet column name)
- 02:59 AM Revision 10113: schemas/util.sql: added mk_source_col(), which uses the schema name instead of the map spreadsheet header to get the datasource name
- 02:44 AM Revision 10112: schemas/util.sql: added table_schema()
- 01:15 AM Revision 10111: added planning/goals/iPlant_BIEN_Proposal_Final.pdf with Mark's e-mail notes in iPlant_BIEN_Proposal_Final.pdf.notes.txt
07/01/2013
06/28/2013
- 04:54 PM Revision 10109: empty inputs/*/import_order.txt: added subdirs in the order they are used by inputs/input.Makefile, by running make on the inputs to auto-populate import_order.txt. import_order.txt is needed by the runscripts to run the right set of subdirs in the right order.
- 04:48 PM Revision 10108: added inputs/.TNRS/grants.sql, with statements to provide SELECT access to bien_read. these statements must be in grants.sql to avoid them being filtered out by pg_dump_limit.
- 04:47 PM Revision 10107: inputs/input.Makefile: added support for separate grants.sql file, which may contain GRANT statements that would normally be filtered out by pg_dump_limit
- 04:44 PM Revision 10106: inputs/input.Makefile: sql/install: added $debug option to run the *.sql import verbosely, to display which statements are being run. this should only be used for SQL files that use COPY FROM to import data, to avoid echoing pages of insert statements.
- 01:53 PM Revision 10105: inputs/input.Makefile: keep $(sortFile) up-to-date: use sort_file_updated=1 flag to indicate that import_order.txt has already been checked, so that recursive invocations of make don't need to recheck it. also use this flag instead of an explicit $(MAKECMDGOALS) list to prevent the $(sortFile) check from being infinite-recursively reinvoked when input.Makefile is read as part of the $(sortFile) check itself.
- 01:38 PM Revision 10104: inputs/input.Makefile: keep import_order.txt up-to-date by running `make $(sortFile)` each time make is run. this ensures that new datasources always have import_order.txt populated when make is first run. eventually, $(tables) can be always set to $(allTables) so that this auto-updating can also be used to ensure that new subdirs added by the user always make it into import_order.txt (so that they will be included in the subdirs that get remade, etc.). import_order.txt is primarily for specifying the order of the subdirs, but some datasources also use it to filter *out* subdirs, so it can't yet be always updated to include the full list of subdirs. however, the filter-out usage should no longer be necessary after the switch to new-style import.
- 12:58 PM Revision 10103: inputs/input.Makefile: added $(filter_make), used to filter the output of embedded $(shell make ...) invocations
- 11:39 AM Revision 10102: inputs/input.Makefile: $(sortFile): use $(filter-out)->then instead of $(filter)->else for clarity
- 11:21 AM Revision 10101: inputs/input.Makefile: added $(sortFile) (import_order.txt) target which adds any missing tables to import_order.txt
- 11:03 AM Revision 10100: inputs/input.Makefile: added list_tables to print $(tables) for use in populating import_order.txt
- 02:50 AM Revision 10099: web/links/index.htm: updated to Firefox bookmarks. grouped version control systems into new version control folder.
06/27/2013
- 09:54 PM Revision 10098: inputs/.NCBI/: added new-style import runscripts, which renamed the staging table columns to VegCore
- 04:48 PM Revision 10097: bugfix: lib/runscripts/datasrc_dir.run, subdir.run: need to remove leading . from dir name to get installed schema name, using new dir2schema()
- 04:47 PM Revision 10096: lib/runscripts/datasrc_dir.run, subdir.run: use new lib/sh/datasrc.sh, which contains code in common to both datasrc-related dir runscripts
- 04:46 PM Revision 10095: added lib/sh/datasrc.sh
- 03:47 PM Revision 10094: inputs/.TNRS/schema.sql: AcceptedTaxon: removed Annotations entry because the accepted name only contains name elements, not additional text (vegpath.org/cf_aff)
- 01:02 PM Revision 10093: bugfix: /README.TXT: Maintenance: syncing ~/bien to ~/Dropbox/svn: added overwrite=1 so that perms transfer from the authoritative ~/bien regardless of relative mtimes
- 12:45 PM Revision 10092: removed no longer used lib/import.sh. use lib/runscripts/table.run instead.
- 12:28 PM Revision 10091: added inputs/*/*/header.csv for CSV inputs, which are now generated by inputs/input.Makefile %/install
- 12:23 PM Revision 10090: added inputs/FIA/*/{VegBIEN.csv,test.xml.ref}, which are now generated by the mapping process for the joined-together tables (even though they are not used by the import, because only occurrence_all is imported)
- 12:20 PM Revision 10089: added inputs/GBIF/_archive/
- 12:18 PM Revision 10088: removed inputs/GBIF/Specimen/, which has been replaced by the refresh in raw_occurrence_record_plants/
- 12:17 PM Revision 10087: added inputs/GBIF/map.csv, used to regenerate inputs/GBIF/raw_occurrence_record_plants/map.csv when raw_occurrence_record_plants is resubset
- 12:12 PM Revision 10086: inputs/FIA/*/postprocess.sql: removed svn:executable attribute using `svn pdel svn:executable ...` now that these are not shell scripts
- 12:11 PM Revision 10085: removed no longer needed inputs/FIA/import. use inputs/FIA/run instead.
- 12:10 PM Revision 10084: inputs/FIA/*/import: changed to postprocess.sql for use by the runscripts
- 04:27 AM Revision 10083: added inputs/FIA/run
- 04:26 AM Revision 10082: added inputs/FIA/*/run. these do not yet use the postprocessing operations in */import.
- 04:24 AM Revision 10081: added inputs/FIA/table.run (for use by table subdirs) and helper Makefile
- 04:17 AM Revision 10080: added lib/runscripts/view.run, for use with table subdirs for views, such as inputs/FIA/occurrence_all/
- 02:14 AM Revision 10079: planning/timeline/timeline.2013.xls: added Reload analytical database checkmark for every Rebuild core database checkmark, because these are always done together as part of the import process
- 01:41 AM Revision 10078: bugfix: inputs/FIA/occurrence_all/import: don't re-prepend * to terms because this is a view, and the underlying columns have already been mapped
- 01:40 AM Revision 10077: bin/src_map: support custom (or no) new_term_prefix. no new_term_prefix is useful for views whose columns have already been renamed in the underlying tables and should not have * re-prepended.
- 01:03 AM Revision 10076: planning/timeline/timeline.2013.xls: moved longer-term goals to new August column, leaving near-term goals in July
- 01:00 AM Revision 10075: planning/timeline/timeline.2013.xls: erased cells where a task was planned but not worked on, so that all shaded cells in the past have check marks to indicate completion of a portion of the task, and empty shaded cells in the future indicate work left to do
- 12:50 AM Revision 10074: planning/timeline/timeline.2013.xls: updated for current progress. renamed "Rerun species range models" to "Prepare to rerun species range models" because the range modeling itself is not part of the BIEN DB development. added a column for July with the tasks that are not yet complete.
06/26/2013
- 06:57 PM Revision 10073: bugfix: inputs/FIA/REF_SPECIES/import: PLANT_SYMBOL_TYPE: prepended * since it's a datasource column, and needs to match up with *PLANT_SYMBOL_TYPE in other table for joins
- 06:57 PM Revision 10072: bugfix: inputs/FIA/REF_SPECIES/import: PLANT_SYMBOL_TYPE: prepended * since it's a datasource column, and needs to match up with *PLANT_SYMBOL_TYPE in other table for joins
- 06:23 PM Revision 10071: schemas/util.sql: try_create(): also ignore wrong_object_type exceptions thrown when trying to alter a view's columns
- 03:36 PM Revision 10070: added inputs/FIA/_src/run, which runs ./download
- 03:00 PM Revision 10069: lib/sh/make.sh: make(): run sys_cmd_path at a higher log_level since the make() steps should not be displayed by default
- 02:58 PM Revision 10068: /README.TXT: to synchronize vegbiendev, jupiter, and your local machine: added step to update mtimes/perms on ~/Dropbox/svn/ so that copying files back to ~/bien does not overwrite the permissions from what is on vegbiendev
- 02:44 PM Revision 10067: inputs/: don't upload test*.xml to jupiter on vegbiendev, because these files are also generated by the full database import but should only be backed up from one source machine, starscream (the Mac)
- 02:26 PM Revision 10066: bin/make: moved $make_filter_active test to lib/sh/make.sh make() so that it's also used when make() is run directly (e.g. in a runscript) rather than via the bin/make wrapper in the PATH
- 02:22 PM Revision 10065: bugfix: lib/sh/make.sh: make(): need to match absolute `make` paths such as /usr/bin/make
- 02:19 PM Revision 10064: lib/sh/util.sh: added self_name alias and use it in self/self_sys
- 02:18 PM Revision 10063: lib/sh/util.sh: added sys_cmd_path() and use it in cmd2sys
- 01:05 PM Revision 10062: bugfix: bin/make: use separate $make_filter_active flag instead of $is_outermost for avoiding duplicate output filtering, so that an outer runscript, which sets $is_outermost but does not activate the make filter, will not prevent the make filter from being activated when make is invoked
- 01:00 PM Revision 10061: bugfix: bin/make: need to use sys_cmd instead of command so that the system make command is invoked instead of the wrapper (which would cause infinite mutual recursion for the ~/bien working copy, although not for the ~/Dropbox/svn working copy because nonrecursive=1 was able to remove the single recursion)
- 12:19 PM Revision 10060: bin/make: use .rel to do relative includes
- 12:19 PM Revision 10059: bugfix: lib/sh/util.sh: .rel(): first use realpath() on BASH_SOURCE[1] in case it's a symlink (as it is for bin/make)
- 12:00 PM Revision 10058: inputs/FIA/_src/Makefile: Extraction: $(zips): use $(allZips) containing a zip for each state so that states that have not yet been downloaded and extracted (or had an empty dir created for them) will be downloaded. previously, the extract target only expanded existing zips but did not download new zips unless no zips had yet been downloaded. (this had been necessary because some states do not have a download, and the download of them would be continuously retried every time the Makefile was run.)
- 11:51 AM Revision 10057: bugfix: inputs/FIA/_src/Makefile: `%: %.zip`: if unzip fails because the download does not exist, create an empty dir for the state instead of aborting make
- 11:33 AM Revision 10056: inputs/FIA/_src/Makefile: use curl instead of wget because that is also available on Mac
- 11:32 AM Revision 10055: bugfix: lib/sh/web.sh: curl(): use --fail so that curl returns a nonzero exit status on error (e.g. file not found) instead of appearing to exit successfully but outputting an error HTML document instead of the file
- 11:05 AM Revision 10054: inputs/FIA/SUBPLOT/map.csv, import: prepended * to all FIA terms to clearly distinguish them from the VegCore terms. this is the standard convention for all datasources, to indicate which terms have not yet been mapped, but was not yet implemented at the beginning of new-style import (the FIA refresh was the first new-style datasource).
- the following replacements were performed to make this change:
in all map.csv: replace regexp (?<=,)(?=[A-Z_]{2,}) wi... - 08:59 AM Revision 10053: inputs/FIA/import_order.txt: added remaining src tables, whose runscripts will be invoked in the order listed by lib/runscripts/datasrc_dir.run
- 08:58 AM Revision 10052: added inputs/FIA/*/_no_import to src tables that are joined together in occurrence_all and should not also be imported separately once they are in import_order.txt
- 07:55 AM Revision 10051: inputs/GBIF/run: inherit from lib/runscripts/datasrc_dir.run, which uses import_order.txt to forward calls to the subdirs
- 07:54 AM Revision 10050: added blank runscripts inputs/GBIF/Source/run, Specimen/run because they are in import_order.txt (used by lib/runscripts/datasrc_dir.run)
- 12:34 AM Revision 10049: bugfix: bin/make: do not alter the PATH passed to the invoked make command, since this is a general-purpose wrapper and is not linked to a specific working copy (it could be used to wrap any make invocation, not just for commands in the svn dir). this uses lib/sh/local.sh's new PATH_add= flag.
- 12:30 AM Revision 10048: lib/sh/local.sh: added PATH_add= flag to allow turning off the addition of $bin_dir_abs to the PATH. this is useful for wrapper scripts that should not alter the PATH passed to their invoked command.
- 12:28 AM Revision 10047: bugfix: lib/sh/make.sh: make(): invoke only the system make command instead of any wrapper for it in the PATH (by using self_sys instead of self), to prevent infinite recursion. single recursion is resolved by nonrecursive=1, but there are cases where mutual recursion occurs due to the presence of two, different bin/makes in the PATH (e.g. if you have two working copies with bin/make, and one is symlinked in your ~/bin/ folder), and these cases can only be resolved by clearing out the PATH completely (since the bin/makes do not know of each other's existence, in order to remove their parent dirs from the PATH).
- 12:23 AM Revision 10046: lib/sh/util.sh: self_sys alias: use new sys_cmd() instead of `command -p` so that only the command path resolution is performed with a limited PATH, and the invoked command itself inherits the full PATH
- 12:22 AM Revision 10045: lib/sh/util.sh: added sys_cmd(), which runs a system command and allows running a system command of the same name as the script
- 12:20 AM Revision 10044: lib/sh/util.sh: added echo_builtin()
06/25/2013
- 06:37 PM Revision 10043: inputs/.rsync_ignore: test*.xml: turn on syncing again, but always treat the local side of the sync (starscream or vegbiendev) as the authoritative copy since they are the machines the tests can be run on
- 05:18 PM Revision 10042: /.rsync_ignore: temp files: hide them on upload so that they are never synced to jupiter. hiding is different than unidirectionally exclude'ing them, because it also causes them to be deleted on the destination if they were uploaded in previous syncs.
- 04:57 PM Revision 10041: inputs/VegBIEN/TWiki/.rsync_ignore: /**: turn syncing back on, but only allow it unidirectionally from vegbiendev->jupiter->starscream to avoid clobbering the live site or the jupiter backup. this is probably the only dir whose authoritative copy is *always* on vegbiendev. for all other dirs, edits can be made wherever convenient, so no copy is authoritative and no sync directions need to be restricted.
- 04:27 PM Revision 10040: /README.TXT: Maintenance: synchronization: fixed whitespace
- 04:07 PM Revision 10039: inputs/.rsync_ignore: install.log.sql: only exclude this on starscream (the local machine), using new machine-specific .rsync_filters, so that vegbiendev's copies of this will be backed up
- 03:46 PM Revision 10038: lib/sh/sync.sh: upload(): .rsync_filter: also support machine-specific filters, for cases when different machines produce the same file (e.g. a log file) but only one machine's copy should be backed up
- 03:43 PM Revision 10037: /README.TXT: Maintenance: to synchronize a Mac's settings with my testing machine's: removed filters that are now handled by .rsync_ignores
- 03:31 PM Revision 10036: added inputs/GBIF/_src/.rsync_filter.upload,download to prevent old versions of GBIFPortalDB-*.dump.gz from being downloaded to the local machine, while keeping them on jupiter. this avoids the need to store these files in ~/Documents/BIEN/large_files/ with symlinks from inputs/GBIF/_src/ to exclude them from the sync.
- 03:17 PM Revision 10035: bugfix: /README.TXT: Maintenance: to synchronize a Mac's settings with my testing machine's: sync ~/Dropbox/svn/ (the no-unversioned-files working copy) separately from the rest of the files, because .svn/ is now excluded by /.rsync_ignore, so that `svn up` needs to be used to keep the .svn/ dirs in sync. note that .svn/ should generally not be synced between machines, because they may use incompatible versions of the svn working copy format.
- 03:02 PM Revision 10034: /README.TXT: Maintenance: to synchronize a Mac's settings with my testing machine's: use new bin/sync_upload (with $sync_remote_subdir) so that per-dir .rsync_ignores are processed, and to use the default $sync_remote_url
- 02:57 PM Revision 10033: lib/sh/local.sh: $sync_remote_url: allow user to override just the sync subdir (not the whole URL) in $sync_remote_subdir. this is useful e.g. for backing up the Mac's files to jupiter.
- 02:28 PM Revision 10032: /README.TXT: Maintenance: to synchronize vegbiendev, jupiter, and your local machine: use new bin/sync_upload instead of specifying all the filter patterns manually. this replaces several `put` commands with various filters with just a bin/sync_upload each on vegbiendev and your machine (in overwrite=1 mode to force a complete sync).
- 02:21 PM Revision 10031: bugfix: backups/.rsync_filter.download: need to prevent existing backups from being deleted on the local side, too, by changing hide patterns to exclude
- 02:11 PM Revision 10030: lib/sh/sync.sh: upload(): make put's $subpath option relative to the currdir instead, like the --include paths. note that $subpath unfortunately can't be used in subdirs at this point because it will cause rsync to ignore the .rsync_ignores and .rsync_filters in parent dirs, including the essential .rsync_ignore in the sync root dir.
- 01:42 PM Revision 10029: /README.TXT: removed unnecessary `env` before kw params, which are treated as such whenever they appear before a command name
- 01:22 PM Revision 10028: bugfix: /README.TXT: updated `make backups/download` to `make backups/<file>/download`
- 01:21 PM Revision 10027: backups/Makefile: upload: use bin/sync_upload
- 01:12 PM Revision 10026: inputs/Makefile: download-logs: use bin/sync_upload like upload/download
- 01:07 PM Revision 10025: bugfix: /README.TXT: `make inputs/upload`, `make inputs/download`: added live=1 so that the sync operation runs rather than previewing what will be synced. removed test=1 because this flag is not used by put.
- 01:00 PM Revision 10024: bugfix: inputs/Makefile: upload, download: need to exclude files in .rsync_ignore, so that large local-only files, such as inputs/GBIF/raw_occurrence_record_plants/table*.tsv, do not have to be synced before `make inputs/upload` can complete (the corresponding .gz gets extracted instead); and deleted temp files in inputs/VegBIEN/TWiki/, such as active sessions, are not added back to the live copy on vegbiendev. previously, fixing this required extracting the rsync command run by `make inputs/upload`, etc. and manually editing it to exclude the applicable .rsync_ignore files, each time `make inputs/upload`, etc. was run (including before every column-based import).
- 12:23 PM Revision 10023: bugfix: bin/make: need to leave bin/, ~/bin/ in the PATH when running make nonrecursively, so that commands invoked by it which are located in these dirs (e.g. put, which will be used by `make inputs/upload`) can still be found. this requires using command()'s new nonrecursive=1 flag instead of running no_PATH_recursion, so that no_PATH_recursion() only affects the resolution of the command path, but does not propagate the filtered PATH to the invoked command itself.
- 12:18 PM Revision 10022: lib/sh/util.sh: command(): added nonrecursive=1 flag, which uses cmd2abs_path to run an external command nonrecursively
- 12:16 PM Revision 10021: lib/sh/util.sh: added cmd2abs_path, which makes the command in $1 nonrecursive
- 11:37 AM Revision 10020: bugfix: lib/sh/util.sh: PATH_rm(): also need to remove adjacent occurrences of the same path (or occurrences which become adjacent when other paths are removed), which :...: matching wasn't doing because the trailing : is consumed, preventing it from being matched at the beginning of the next path. since unlike filesystem paths with /, it is not necessary for a match to span multiple :-separated sections, we can just use new split() to split apart the PATH into an array of paths, filter each path, and join() them back together.
- 11:33 AM Revision 10019: lib/sh/util.sh: added split()
- 10:32 AM Revision 10018: lib/sh/util.sh: auto-echo common external commands: added `which`
- 10:32 AM Revision 10017: lib/sh/util.sh: auto-echo common external commands: use simpler echo_run instead of command since logging handling is not needed
- 08:50 AM Revision 10016: added backups/vegbien.r9897.backup.md5
06/23/2013
- 09:30 PM Revision 10015: lib/sh/sync.sh: upload(): documented that each --include path is relative to the *currdir*, not the root dir of the upload ($local_dir). this feature, although previously unintended, is actually better because the user can change to a subdir of the root dir and specify upload paths relative to the dir they are in. however, when invoking upload() from a script with --include paths specified, this means you need to use an absolute path (e.g. "$(dirname "${BASH_SOURCE[0]}")"/...; or the value that will become $local_dir, which for sync_upload() is $root_dir).
- 08:56 PM Revision 10014: backups/.rsync_ignore: replaced with .rsync_filter.upload to allow uploading new backups but not deleting existing backups if they don't exist on the local (rsync-invoking) side; and .rsync_filter.download to avoid downloading backups to the local side. this allows storing older backups just on jupiter, where there is much more disk space. note that this change must be made on the *remote* side (jupiter) for it to be effective, because these are remote-side rules and are only processed by the remote-side rsync instance.
- 08:55 PM Revision 10013: lib/sh/sync.sh: upload(): use directional .rsync_filter to supplement .rsync_ignore with all kinds of --filter rules. separate .rsync_filters are needed for the upload (swap=) and download (swap=1) directions because the sender and the receiver are reversed, causing asymmetric rules like protect/hide to change meaning.
- 07:48 PM Revision 10012: updated backups/TNRS.backup.md5
- 07:48 PM Revision 10011: added backups/TNRS.2013-6-17.backup.md5, TNRS.2013-6-22.backup.md5
- 03:58 PM Revision 10010: /README.TXT: Backups: TNRS cache: Back up/Restore: added runtimes (3 min/5.5 min)
- 03:52 PM Revision 10009: lib/sh/sync.sh: upload(): usage: documented put's swap=1 flag, which downloads instead of uploads
- 03:47 PM Revision 10008: added inputs/GBIF/raw_occurrence_record_plants/.rsync_ignore with filters that have previously needed to be manually added whenever `make inputs/upload` was run
- 03:46 PM Revision 10007: added inputs/GBIF/_MySQL/.rsync_ignore with filters from /README.TXT > Maintenance > to synchronize vegbiendev, jupiter, and your local machine. these filters will now be used with bin/sync_upload in addition to the periodic backup commands.
- 03:45 PM Revision 10006: added inputs/VegBIEN/TWiki/.rsync_ignore with filters from /README.TXT > Maintenance > to synchronize vegbiendev, jupiter, and your local machine. these filters will now be used with bin/sync_upload in addition to the periodic backup commands.
- 03:44 PM Revision 10005: added inputs/.rsync_ignore with filters from inputs/Makefile $(rsyncSrcs). these filters will now be used with bin/sync_upload in addition to `make inputs/upload`.
- 03:43 PM Revision 10004: added bin/.rsync_ignore with filters from /README.TXT > Maintenance > to synchronize vegbiendev, jupiter, and your local machine. these filters will now be used with bin/sync_upload in addition to the periodic backup commands.
- 03:40 PM Revision 10003: added backups/.rsync_ignore with filters from /README.TXT > Maintenance > to synchronize vegbiendev, jupiter, and your local machine. these filters will now be used with bin/sync_upload in addition to the periodic backup commands.
- 03:36 PM Revision 10002: /.rsync_ignore: added *.pyc
- 03:36 PM Revision 10001: added /.rsync_ignore with filters from lib/common.Makefile $(rsync). these filters will now be used with bin/sync_upload in addition to `make inputs/upload`.
- 03:34 PM Revision 10000: lib/sh/sync.sh: upload(): use --exclude filters from per-dir .rsync_ignore. note that --exclude-from can't be used for this, because it is relative to the currdir, not the rsync root, and therefore also requires the .rsync_ignore to exist rather than using it only if it exists.
06/22/2013
06/21/2013
06/20/2013
- 10:58 PM Revision 9997: web/links/index.htm: updated to Firefox bookmarks. sorted NCEAS bookmarks to put homepage and support pages first.
- 06:21 PM Revision 9996: /README.TXT: Full database import: To run TNRS, etc. after the main import: clarified that you should only run `export version=<version>` if the import is named something *other than* public (i.e. it has not yet replaced the previous public schema)
- 06:14 PM Revision 9995: /README.TXT: Full database import: To run TNRS: removed `by_col=1` because by-column mode is not applicable to running TNRS. it is, however, needed when running import_scrub (i.e. `make inputs/<datasrc>/reimport_scrub by_col=1`).
- 06:10 PM Revision 9994: inputs/.TNRS/schema.sql: tnrs: vegbiendev update steps: added `make backups/TNRS.backup-remake` to back up TNRS before making changes to it. this provides a more recent restore point than the last import in case the changes mess things up. (however, the last import's backup is usually sufficient unless TNRS has been run since then.)
- 05:53 PM Revision 9993: inputs/.TNRS/schema.sql: tnrs_populate_fields(): added VACUUM ANALYZE and runtime (50 s)
- 05:42 PM Revision 9992: inputs/.TNRS/schema.sql: tnrs_populate_fields(): updated runtime (16 min)
- 05:09 PM Revision 9991: schemas/VegBIEN/taxonomy/higherPlantGroup.xlsx.src.txt: added Brad's comment that there are some holes in the Embryophyte subclasses list, and we need to validate it
- 04:49 PM Revision 9990: inputs/.TNRS/schema.sql: tnrs: documented that when changing this table's schema, you must also make the same changes on vegbiendev. included sample util.set_col_types() call with runtime (4 min).
- 03:58 PM Revision 9989: inputs/.TNRS/schema.sql: tnrs_populate_fields(): updated runtime (16 min)
- 03:32 PM Revision 9988: bugfix: inputs/.TNRS/schema.sql: tnrs_populate_fields(): need to schema-qualify invoked functions
- 03:29 PM Revision 9987: bugfix: inputs/.TNRS/schema.sql: tnrs_populate_fields(): Is_homonym: use the *_is_homonym flag for whichever of genus or family (in that order) is NOT NULL, rather than horizontal-ORing potentially NULL values together
- 03:22 PM Revision 9986: bugfix: inputs/.TNRS/schema.sql: family_is_homonym(), genus_is_homonym(): need to return NULL instead of false when input family/genus is NULL. EXISTS() does not support this, so STRICT is used to provide this functionality automatically.
- 03:19 PM Revision 9985: inputs/.TNRS/schema.sql: added family_is_homonym(), genus_is_homonym() and use them in tnrs_populate_fields()
- 03:15 PM Revision 9984: inputs/.TNRS/schema.sql: score_ok(): changed to IMMUTABLE and STRICT
- 03:14 PM Revision 9983: inputs/.TNRS/schema.sql: tnrs_populate_fields(): updated runtime (16 min)
- 02:41 PM Revision 9982: inputs/.TNRS/schema.sql: tnrs_populate_fields(): never_homonym: use Author_score threshold to exclude matches that are too fuzzy to confirm the presence of a plant name author
- 02:38 PM Revision 9981: bugfix: inputs/.TNRS/schema.sql: tnrs_populate_fields(): *_is_homonym: also need to check that there was no Author_matched (i.e. that it could be a homonym). Is_homonym: use new never_homonym var.
- 02:18 PM Revision 9980: inputs/.TNRS/schema.sql: tnrs_populate_fields(): updated runtime (18 min)
- 02:17 PM Revision 9979: inputs/import.stats.xls: Updated import times
- 02:07 PM Revision 9978: planning/workflow/bien3_architecture.pptx: stage II: removed Step prefix before stage #, which the other slides don't have
- 01:57 PM Revision 9977: added planning/workflow/bien3_architecture/stages.png
- 01:36 PM Revision 9976: added planning/workflow/bien3_architecture/stage_*.png
- 12:54 PM Revision 9975: added planning/workflow/bien3_architecture.pptx
- 08:20 AM Revision 9974: inputs/.TNRS/schema.sql: tnrs_populate_fields(): when changing this function: UPDATE statement: include TNRS schema since it may not be in the search_path
- 08:14 AM Revision 9973: inputs/.TNRS/schema.sql: tnrs_populate_fields(): Is_plant: also consider homonyms using new family_is_homonym, genus_is_homonym (see wiki.vegpath.org/Result_filtering#taxon_is_plant)
- 08:03 AM Revision 9972: inputs/.TNRS/schema.sql: tnrs: added Is_homonym derived col (uses IRMNG.family_homonym_epithet, genus_homonym_epithet)
- 07:54 AM Revision 9971: schemas/vegbien.sql: re-ran `make schemas/public/reinstall; make schemas/remake` cycle, which apparently changed sort order of statements
- 07:11 AM Revision 9970: /README.TXT: Full database import: disk space check: updated minimum (to 300GB) for new import schema size. note that most of the space (166GB) is indexes, and even of the 87GB of data, only 20GB is from GBIF and 15GB from FIA (so most of it is duplication).
- 07:07 AM Revision 9969: added inputs/IRMNG/*_homonym_epithet/map.csv, etc. (created by */run)
- 07:01 AM Revision 9968: bugfix: inputs/input.Makefile: `%/install %/header.csv: %/create.sql`: in noclobber mode, mark %/header.csv as .PRECIOUS so the existing file won't be deleted if the table already exists (causing an error exit)
- 06:54 AM Revision 9967: bugfix: lib/runscripts/table.run: remake_VegBIEN_mappings(): run yes using piped_cmd() so the SIGPIPE doesn't cause an errexit
- 06:45 AM Revision 9966: added inputs/IRMNG/{genus_homonym_epithet,family_homonym_epithet}/run, which inherit from ../table.run so that load_data() (which runs create.sql) is invoked
- 06:44 AM Revision 9965: added inputs/IRMNG/species_homonyms/new_terms.csv
- 06:43 AM Revision 9964: bugfix: added no-op inputs/IRMNG/Source/run so inputs/IRMNG/run would have something to invoke for it
- 06:35 AM Revision 9963: inputs/IRMNG/run: use lib/runscripts/datasrc_dir.run, which now provides import() and $subdirs
- 06:34 AM Revision 9962: lib/runscripts/datasrc_dir.run: extend import.run and provide an import() implementation that runs all the runscripts for import_order.txt subdirs
- 06:20 AM Revision 9961: lib/csvs.py: sniff(): support single-column spreadsheets by defaulting to the Excel dialect when the delimiter can't be determined
- 06:09 AM Revision 9960: inputs/IRMNG/: added family_homonym_epithet, genus_homonym_epithet lookup tables, which use util.all_same() to filter out internal Plantae homonyms
- 05:44 AM Revision 9959: schemas/util.sql: added all_same() aggregate
- 05:31 AM Revision 9958: schemas/util.sql: added not_empty(anyarray)
- 12:14 AM Revision 9957: schemas/util.sql: added not_null() (usable as an aggregate's FINALFUNC)
- 12:13 AM Revision 9956: schemas/util.sql: added not_null() (usable as an aggregate's FINALFUNC)
06/19/2013
- 10:18 PM Revision 9955: bugfix: inputs/IRMNG/import_order.txt: need to specify order so that Source is first
- 10:16 PM Revision 9954: bugfix: inputs/IRMNG/*/map.csv: remapped Authority to scientificNameAuthorship instead of authors (now data_authors <VegCore.vegpath.org?data_authors> for clarity)
- 09:59 PM Revision 9953: inputs/IRMNG/map.csv: updated to scrubbed output names from */map.csv (/map.csv does not currently get scrubbed)
- 09:51 PM Revision 9952: bugfix: inputs/IRMNG/species_homonyms/header.csv, map.csv: reset input columns to DSV (delim-separated values) header. they had gotten changed to the output names in running map.csv with remake=1, causing it to be remade from the (renamed) staging tables.
- 08:54 PM Revision 9951: inputs/input.Makefile: $(_svnFilesGlob): added *Makefile
- 08:51 PM Revision 9950: /README.TXT: `make inputs/{upload,download}`: first run with test=1 to see what the diffs will be
- 08:47 PM Revision 9949: added inputs/IRMNG/, including runscripts to download the names. this is now the 2nd datasource after GBIF to use runscripts, and the 3rd after FIA/GBIF to use new-style import.
- 08:45 PM Revision 9948: inputs/input.Makefile: $(_svnFilesGlob): added *run (runscripts)
- 08:24 PM Revision 9947: lib/runscripts/table.run: import(): also run remake_VegBIEN_mappings() to accept the test output. this function was previously unused, but was left in for future use when lib/import.sh was translated to lib/runscripts/table.run (it was used in its import.sh form in inputs/FIA/occurrence_all/import).
- 08:21 PM Revision 9946: bugfix: lib/runscripts/table.run: remake_VegBIEN_mappings(): need to change to $top_dir before running `rm header.csv map.csv`
- 08:18 PM Revision 9945: lib/sh/util.sh: added in_top_dir()
- 08:00 PM Revision 9944: lib/runscripts/table.run: remake_VegBIEN_mappings(): only remake header.csv, map.csv if this target is being run directly, to avoid needing to remake them every time. for tables that are views, this instead requires them to be explicitly remade when the view columns change.
- 07:07 PM Revision 9943: bugfix: lib/runscripts/subdir.run: subdir_make(): only remake if $remake has been explicitly propagated to subdir_make() by using self_make
- 06:51 PM Revision 9942: lib/sh/make.sh: added deferred_check_target_exists alias and use it in check_fake_target_exists
- 06:41 PM Revision 9941: added lib/sh/web.sh with curl wrapper
- 06:41 PM Revision 9940: lib/sh/make.sh: added check_wildcard_target_exists alias
- 06:37 PM Revision 9939: lib/sh/util.sh: added wildcard1 alias
- 06:37 PM Revision 9938: lib/sh/util.sh: added echo1()
- 06:32 PM Revision 9937: lib/runscripts/table.run: load_data(): first make sure schema is installed
- 06:31 PM Revision 9936: lib/runscripts/table.run: added datasrc_make_install()
- 06:31 PM Revision 9935: table_make_install(): take $install_log as an overridable kw param to support install logs in different locations
- 05:55 PM Revision 9934: lib/runscripts/table.run: load_data(): split noclobber functionality into separate table_make_install() function, which can be used by other install-related targets
- 03:27 PM Revision 9933: added schemas/VegBIEN/taxonomy/higherPlantGroup.xlsx.src.txt with Brad's description of how the names were chosen
- 02:37 PM Revision 9932: added schemas/VegBIEN/taxonomy/higherPlantGroup.xlsx
- 01:50 PM Revision 9931: schemas/VegBIEN/planning/taxonomy/: moved non-VegBIEN-specific resources to planning/resources/taxonomy/. this includes Brad's all-important Nomenclature_excerpt.ppt with the Latin taxonomic hierarchy suffixes on slide 5.
- 11:02 AM Revision 9930: bugfix: schemas/vegbien.sql: taxon_trait_view: use the TNRS-scrubbed name from ScrubbedTaxon when available
- 10:36 AM Revision 9929: schemas/vegbien.sql: split geoscrub_input_view's new-row-only filtering into separate view geoscrub_input_new, so that the full geoscrub_input rows are still available. the reduction in geoscrub_input from eliminating the already-scrubbed rows was only 280,000 (5076500 - 4799173) out of a possible 1.7 million (1707970), so it makes sense to just run geoscrubbing on the full input. (the lower-than-expected reduction is most likely due to rows from pre-refresh data being present in the original geoscrub_output table, which have been replaced by different, post-refresh input rows.)
- 10:18 AM Revision 9928: added exports/_archive/
- 10:17 AM Revision 9927: mappings/VegCore-VegBIEN.csv: genus->taxonlabel.taxonomicname: use new _filter_genus() (see r9882)
- 10:15 AM Revision 9926: backups/TNRS.backup.md5: updated
- 09:43 AM Revision 9925: bin/make_analytical_db: use new mk_table() instead of TRUNCATE/INSERT
- 09:41 AM Revision 9924: bin/make_analytical_db: added mk_table() and use it in mk_analytical_table()
- 09:30 AM Revision 9923: schemas/vegbien.sql: higher_plant_group_nodes: ferns and allies: added Lycopodiophyta node, as requested by Brad in the conference call (wiki.vegpath.org/2013-06-13_conference_call)
- 09:30 AM Revision 9922: schemas/vegbien.sql: geoscrub_input_view: exclude rows that have already been geoscrubbed, by anti-joining on geoscrub_output
- 09:11 AM Revision 9921: inputs/.geoscrub/geoscrub_output/postprocess.sql: set decimallatitude, decimallongitude types to double precision to facilitate joining with other double precision values
- 09:02 AM Revision 9920: inputs/.geoscrub/geoscrub_output/postprocess.sql: coords index: added rest of input columns so this can be used to check the existence of a result by input. added runtime (55 s). use idempotent create_if_not_exists().
- 08:17 AM Revision 9919: schemas/vegbien.sql: higher_plant_group_nodes: ferns and allies: added Lycopodiophyta node, as requested by Brad in the conference call (wiki.vegpath.org/2013-06-13_conference_call)
- 08:08 AM Revision 9918: bugfix: schemas/vegbien.sql: higher_plant_group_nodes: removed ferns and allies nodes Anthocerotophyta, Marchantiophyta, Bryophyta, which were incorrectly said to be part of this clade in the BIEN2 analytical DB overview (/planning/workflow/validation/BIEN2_Analytical_DB_overview.docx > p. 13 bottom > last ¶). see http://wiki.vegpath.org/2013-06-13_conference_call#fix-higher_plant_group_nodes-mapping .
- 07:58 AM Revision 9917: bugfix: /Makefile: postgres-Linux: phpPgAdmin: added steps to configure it for Apache 2.4
06/18/2013
- 07:49 PM Revision 9916: /run: geoscrub_input/make(): documented runtime (40 s)
- 06:22 PM Revision 9915: bin/make_analytical_db: added `/run export_` to make the geoscrub_input CSV export
- 06:21 PM Revision 9914: inputs/.TNRS/schema.sql: tnrs_populate_fields(): removed no longer needed casts of *_score to double precision
- 06:06 PM Revision 9913: inputs/.TNRS/schema.sql: tnrs: *_score: changed type to double precision because these fields are always floats. this also avoids the need to manually cast them to double precision each time they are used.
- 05:55 PM Revision 9912: lib/tnrs.py: HTTP requests: rewrapped lines
- 05:53 PM Revision 9911: lib/tnrs.py: updated HTTP requests to match current web app
- 05:51 PM Revision 9910: bugfix: lib/tnrs.py: download_request_template: changed dirty to true (to match the current web app), which is apparently needed to apply the source_sorting setting to the downloaded TSV in addition to the GUI results
- 05:29 PM Revision 9909: lib/tnrs.py: retrieval_request_template: turned source_sorting back off, because it causes any match from the first source to always be used, even if it has a lower match score than the match from the other source. (Brad confirms that this should be off.) I think we had this on originally to ensure that only Tropicos results were used when available, rather than USDA when it was a better match. *** note that due to a bug in the web app, this change will not actually be effective, because the source_sorting option is only applied to the GUI results, not the downloaded TSV. ***
- 04:27 PM Revision 9908: inputs/.TNRS/schema.sql: tnrs: Name_number: changed type to integer so it would sort numerically
- 04:24 PM Revision 9907: inputs/.TNRS/schema.sql: added pkey on Time_submitted, Name_number
- 04:21 PM Revision 9906: inputs/.TNRS/schema.sql: changed Name_submitted pkey to a unique constraint to allow adding a pkey on Time_submitted, Name_number instead
- 04:14 PM Revision 9905: inputs/.TNRS/schema.sql: Time_submitted, Name_number: added NOT NULL constraints so that they can be used in a unique constraint
- 02:21 PM Revision 9904: lib/tnrs.py: submission_request_template: include GCC in addition to Tropicos, because it provides more synonyms than Tropicos for Asteraceae, and the accepted names still match the Tropicos backbone (https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/2013-06-13_conference_call#include-GCC-when-running-TNRS)
06/17/2013
- 08:18 PM Revision 9903: inputs/.TNRS/tnrs/tnrs.make: removed no longer needed end time, now that the total runtime is printed
- 08:17 PM Revision 9902: inputs/.TNRS/tnrs/tnrs.make: print the total runtime using `time`
- 08:14 PM Revision 9901: inputs/.TNRS/tnrs/tnrs.make: include the end time in addition to the start time so that the total runtime can be calculated
- 07:59 PM Revision 9900: lib/sh/util.sh: command-specific alternate stdin/stdout/stderr: choice of 40/41/42: added mnemonic that 4 looks like A for Alternate
06/14/2013
- 09:26 AM Task #636 (New): support TNRS-matching names in "", insted of encoding them and letting TNRS return no match
- * names like @"Acer rubrum"@ (_with_ quotes) currently get encoded as @ !quo Acer rubrum !quo @, which TNRS can't par...
- 07:42 AM Revision 9899: bugfix: /README.TXT: Full database import: added step to remove any leftover TNRS lockfile. usually, the PID in it would not exist, but sometimes it now refers to a different, active process which blocks tnrs.make.
06/13/2013
- 01:00 PM Revision 9898: schemas/vegbien.sql: allow public_ to view lookup tables (cultivated_family_locations, higher_plant_group_nodes)
- 08:13 AM Revision 9897: added backups/TNRS.backup.md5, vegbien.r9459.backup.md5
06/12/2013
- 02:45 PM Revision 9896: bugfix: lib/sh/local.sh: sync_upload(): need to use --no-group to prevent the group from being reset to aaronmk upon download from jupiter (which uses group aaronmk instead of bien). use ./fix_perms to set the group of all files to bien. also use --no-owner in case running as root.
- 02:12 PM Revision 9895: lib/sh/sync.sh: removed sync_download(). use swap=1 sync_upload() instead.
- 02:11 PM Revision 9894: lib/sh/sync.sh: removed download(). use swap=1 upload, or swap=1 upload_caller, instead. this avoids having separate upload()/download() pairs for every caller of upload(), because you can instead just set swap=1.
- 01:50 PM Revision 9893: bugfix: lib/sh/sync.sh: upload(): don't kw_params $swap because this unexports it, preventing put from seeing it. instead, use echo_vars to print it.
- 01:35 PM Revision 9892: added bin/sync_upload, a wrapper around sync_upload()
- 01:23 PM Revision 9891: added bin/sync_upload, a wrapper around sync_upload()
- 01:00 PM Revision 9890: bugfix: lib/sh/sync.sh: upload(): only add `--exclude="**"` if there are --includes. this enables running upload() without paths to upload all files.
- 12:56 PM Revision 9889: lib/sh/sync.sh: upload(): support passing -- options to put, which will not be run through the path->--include processing
- 12:55 PM Revision 9888: bugfix: lib/sh/sync.sh: upload(): added missing `local args=()` initializer
- 12:18 PM Revision 9887: /README.TXT: Full database import: On local machine: added step to do steps under Maintenance > "to synchronize vegbiendev, jupiter, and your local machine", which is needed in addition to `make inputs/upload` since that doesn't handle overwrites or deletions
- 12:10 PM Revision 9886: /README.TXT: Maintenance: to synchronize vegbiendev, jupiter, and your local machine: added warning that you should pay careful attention to all files that will be deleted or overwritten (as the three machines are often out of sync)
- 11:26 AM Revision 9885: added inputs/GBIF/_MySQL/GBIFPortalDB-2013-02-20.data.0.preamble.sql
- 11:17 AM Revision 9884: /README.TXT: Full database import: make inputs/{upload,download}: run them first with `test=1` to see what the changes will be
- 11:12 AM Revision 9883: /README.TXT: Full database import: `svn up`: use --force to avoid errors about existing files
- 10:49 AM Revision 9882: mappings/VegCore-VegBIEN.csv: genus->taxonlabel.taxonomicname: filter out genera that contain numbers (using new _filter_genus()), which break TNRS and prevent it from matching any other parts of the name. later, these genera can instead be moved to the end of the name, where TNRS will correctly match them as Unmatched_terms.
- 10:48 AM Revision 9881: bugfix: inputs/VegBIEN/: added _no_import to disable import for this folder, since this is actually just an entry in web/datasources/ with VegPath redirection links, rather than an input to the import process. this fixes "schema "VegBIEN" does not exist" errors generated in `make test`.
- 10:45 AM Revision 9880: inputs/input.Makefile: $(dontImport): also support putting a _no_import file at the top level in the datasource to exclude the entire datasource
- 10:30 AM Revision 9879: bugfix: lib/sh/local.sh: removed make() override, which is no longer needed now that its operations are performed by verbosity_compat(), and which caused errors by setting $verbosity to the invalid value ""
- 10:28 AM Revision 9878: bugfix: lib/sh/util.sh: verbosity_compat(): always use default verbosity (`unset verbosity`) when verbosity == 1, regardless of whether the caller has set $verbosity to the special value "", because $verbosity is supposed to be an integer field and "" is not supported by most functions that use $verbosity. in cases where a util.sh script is invoked, it will set $verbosity back to the default value 1, so this will function as before for util.sh scripts and fix $verbosity for scripts that use a different verbosity system.
- 10:05 AM Revision 9877: added inputs/GBIF/raw_occurrence_record_plants/table.tsv.md5
- 09:51 AM Revision 9876: inputs/GBIF/raw_occurrence_record_plants/test.xml.ref: regenerated. updated for new staging table input columns, which are now the same as the output columns.
- 09:41 AM Revision 9875: bugfix: inputs/input.Makefile: %/VegBIEN.csv: use header from map.csv instead of the new columns, so that source.shortname is set to GBIF instead of VegCore
- 09:24 AM Revision 9874: inputs/input.Makefile: %/VegBIEN.csv: when a runscript is available, instead map the *output* columns of map.csv to VegBIEN, because the columns have been renamed in the staging table
- 08:32 AM Revision 9873: inputs/GBIF/raw_occurrence_record_plants/VegBIEN.csv: regenerated, which adds row_num input col
- 08:16 AM Revision 9872: lib/sh/util.sh: echo_func(): check can_log at beginning of function, so that the resource-intensive func_loc (which calls `readlink -f`) does not need to be called if echo_cmd would not log anything at the current verbosity
- 08:02 AM Revision 9871: lib/sh/util.sh: echo_func(): removed no longer used $minor flag. use `clog++... echo_func` instead.
- 07:25 AM Revision 9870: lib/sh/util.sh: verbosity_compat(): don't make $verbosity a local var of this function, because then the changes will not be visible to the caller
- 07:12 AM Revision 9869: bugfix: bin/make: use verbosity_compat because some make-invoked commands (e.g. bin/map) don't support verbosity=""
- 07:10 AM Revision 9868: lib/sh/util.sh: command(): command__exec(): use verbosity_compat to support commands that don't support verbosity=""
- 07:09 AM Revision 9867: lib/sh/util.sh: added verbosity_compat(), for use with commands that don't support verbosity=""
- 06:47 AM Revision 9866: bugfix: lib/sh/local.sh: make(): when invoking overridden func, need make__make_sh
- 06:46 AM Revision 9865: bugfix: lib/sh/util.sh: self, self_sys aliases: need to remove any func_override suffix __* from the FUNCNAME
- 06:35 AM Revision 9864: bugfix: inputs/GBIF/import_order.txt, run: updated raw_occurrence_record/ to raw_occurrence_record_plants/
- 06:32 AM Revision 9863: inputs/FIA/occurrence_all/test.xml.ref: update inserted row count
- 06:27 AM Revision 9862: bugfix: bin/make: include local.sh so that its default verbosity-setting make() override will be used
- 06:26 AM Revision 9861: lib/sh/local.sh: added make() override, which uses the default verbosity (i.e. verbosity="") when verbosity == 1. scripts that use lib/sql.py (which uses $verbosity) have different default verbosities, and this default should not be overriden by an env var, unless a higher verbosity has been set.
- 06:24 AM Revision 9860: lib/sh/local.sh: added missing include of make.sh, used by root_make()
- 05:57 AM Revision 9859: schemas/vegbien.sql: added _filter_genus()
- 04:47 AM Revision 9858: inputs/GBIF/raw_occurrence_record_plants/run: import() runtime: specified that this does not include table.tsv.gz/make()
- 04:07 AM Revision 9857: inputs/GBIF/raw_occurrence_record_plants/postprocess.sql: Remove institutions that we have direct data for: # duplicates: added revision #
- 04:07 AM Revision 9856: inputs/GBIF/raw_occurrence_record_plants/postprocess.sql: Remove institutions that we have direct data for: documented that there are 4.5 million duplicates (59,998,354 rows before - 55,417,646 rows after = 4,580,708)
- 03:49 AM Revision 9855: inputs/GBIF/raw_occurrence_record_plants/postprocess.sql: Remove institutions that we have direct data for: added rerun time (~0 thanks to index, so no problem doing the DELETE each time postprocess.sql is run)
- 03:25 AM Revision 9854: *{.sh,run}: use simpler .rel() instead of `. "$(dirname "${BASH_SOURCE[0]}")"/...` for relative includes
- 03:24 AM Revision 9853: lib/sh/util.sh: added .rel()
- 02:52 AM Revision 9852: schemas/VegCore/VegCore.pg.sql: regenerated from installed schema. a Linux bug that caused constraints to be output in reverse order has now been fixed, as of 12.04.2 (broken in 12.04.1).
- 02:48 AM Revision 9851: bugfix: inputs/GBIF/_MySQL/MySQL_schema, MySQL_data: sed: put {} commands on their own line to work on Mac
- 02:11 AM Revision 9850: lib/sh/util.sh: .(): put echo_func on its own line for clarity
- 02:10 AM Revision 9849: lib/sh/util.sh: .(): added echo_func (2 log_levels up because it's internal)
- 02:06 AM Revision 9848: schemas/VegCore/mk_derived: use new lib/sh/local.sh instead of lib/import.sh (a precursor to util.sh, etc. still used by inputs/FIA/)
06/11/2013
- 07:50 PM Revision 9847: bugfix: load_data(): verbosity_min: use verbosity_min='' so that csv2db's default verbosity (3) is used, instead of setting the verbosity directly to 3, which caused the log++ logging output from bin/make to be echoed at verbosity 3, creating cluttered output
- 07:43 PM Revision 9846: lib/sh/util.sh: verbosity_min(): support value '', which sets verbosity=''
- 06:40 PM Revision 9845: bugfix: inputs/GBIF/raw_occurrence_record_plants/postprocess.sql: updated column names to match the renamings in map.csv, which are now performed on the staging table itself
- 06:38 PM Revision 9844: lib/sh/util.sh: run_args_cmd(): time the command so that the runtime of the outer runscript target (i.e. the command run from the shell) is printed at the end of the output, like in bin/make
- 05:56 PM Revision 9843: bugfix: inputs/input.Makefile: %/install: don't run $(cleanup) if it has already been run by $(import_install_), so that it doesn't run twice
- 05:54 PM Revision 9842: inputs/input.Makefile: %/postprocess: don't run postprocess.sql if it is supposed to be run by a runscript, because postprocess.sql may then depend on additional steps the runscript runs before it
- 05:25 PM Revision 9841: lib/runscripts/table.run: import(): use self_make on load_data so that the remake status determines whether the table is reinstalled
- 05:23 PM Revision 9840: bugfix: lib/runscripts/mysql.table.run: import(): added missing set_make_vars, needed by self_make
- 05:21 PM Revision 9839: bugfix: lib/runscripts/table.run: load_data(): need to use $_remake instead of $remake when using set_make_vars
- 05:20 PM Revision 9838: lib/runscripts/table.run: added set_make_vars to all make targets so $remake would be propagated appropriately
- 05:18 PM Revision 9837: lib/runscripts/table.run: load_data(): also clobber install log if remaking, because the table will be reinstalled
- 05:12 PM Revision 9836: lib/runscripts/table.run: load_data(): automatically select noclobber mode depending on whether the install log already exists. this removes the need for a separate load_data_first_run() function.
- 04:59 PM Revision 9835: bugfix: lib/runscripts/table.run: load_data(): ignore errors if table already exists
- 04:52 PM Revision 9834: lib/runscripts/table.run: load_data(): use noclobber=1 to avoid overwriting the install log when re-running the install target idempotently. load_data_first_run() is now available to preserve the output in the log on the first run.
- 04:51 PM Revision 9833: inputs/input.Makefile: Staging tables installation: $(logInstall): don't output to the install log if $noclobber flag is set, to prevent overwriting the log when re-running the install target idempotently
- 04:18 PM Revision 9832: bugfix: lib/runscripts/mysql.table.run: import(): move previous versions of table.tsv out of the main dir before loading staging tables, to prevent them from being considered a TSV segment file and prepended to table.tsv
- 04:15 PM Revision 9831: lib/sh/util.sh: added mv2dir(), mv_glob which wrap mv
- 04:14 PM Revision 9830: lib/sh/util.sh: added mkdir alias which adds -p to prevent errors if the dir already exists
- 03:45 PM Revision 9829: lib/sh/util.sh: added wildcard alias, similar to make's $(wildcard) function
- 03:29 PM Revision 9828: bugfix: inputs/GBIF/raw_occurrence_record_plants/postprocess.sql: institution_code index: create it idempotently using create_if_not_exists() and an explicit index name, so that a duplicate index doesn't get added each time postprocess.sql is run
- 03:27 PM Revision 9827: lib/sh/local.sh: psql(): don't put util in the search_path because psql scripts now add it themselves if they need it, using `SELECT util.search_path_append(util);`
- 03:22 PM Revision 9826: inputs/GBIF/raw_occurrence_record_plants/postprocess.sql: add util to the search_path so that postprocess.sql will also work when run by inputs/input.Makefile, which only puts the datasource (GBIF) in the search_path
- 03:18 PM Revision 9825: schemas/util.sql: added search_path_append()
- 03:12 PM Revision 9824: schemas/util.sql: added eval() to allow running EXECUTE outside of a function (and to echo the command that is run)
- 09:04 AM Revision 9823: inputs/GBIF/raw_occurrence_record_plants/run: added import() runtime (5 h)
06/10/2013
- 11:58 PM Revision 9822: inputs/GBIF/raw_occurrence_record_plants/run: table.tsv.gz/make() runtime: noted that this excludes the upload time
- 11:58 PM Revision 9821: inputs/GBIF/raw_occurrence_record_plants/run: added table.tsv.gz/upload() runtime (15 min)
- 11:48 PM Revision 9820: added lib/runscripts/mysql.table.run (general to all MySQL datasources) and use it in inputs/GBIF/table.run
- 11:13 PM Revision 9819: inputs/GBIF/raw_occurrence_record_plants/run: table.tsv/make(): to view runtime when using `screen`: keys used to scroll: added Ctrl-B/Ctrl-F for page-at-a-time scrolling (there are a lot of pages of output for the import() target!)
06/09/2013
- 09:21 PM Revision 9818: bugfix: inputs/GBIF/table.run: table.tsv.gz/make(): don't run table.tsv.gz/upload in test mode, to avoid clobbering the backup of a full table.tsv with a partial, testing table.tsv
- 09:18 PM Revision 9817: lib/sh/db.sh: set test mode when using limited # rows
- 09:08 PM Revision 9816: bugfix: inputs/GBIF/table.run: table.tsv.gz/upload(): don't use inplace mode because it leaves a newer mtime when aborted, causing rsync to think that the partial upload is actually newer than the source. note that rsync's --partial-dir mode is just as capable of resuming an aborted upload (it will just use a file in .rsync-tmp instead). inplace mode is primarily designed for fixed-offset files which don't change much between edits, but this is not true for exports (or the gzips of them), which will change the file offsets of most data if even one row or column is added or removed.
- 09:01 PM Revision 9815: bugfix: inputs/GBIF/table.run: table.tsv.gz/make(): run table.tsv.gz/upload here instead of in table.tsv/make() because it should not run until table.tsv.gz is finished being made, which is not the case in table.tsv/make() because table.tsv.gz/make is run in the background
- 08:59 PM Revision 9814: inputs/GBIF/table.run: table.tsv.gz/upload(): moved before table.tsv.gz/make() so it can be used by it
- 08:39 PM Revision 9813: bugfix: inputs/GBIF/table.run: table.tsv.gz/upload(): need overwrite=1 because the mtime of an aborted inplace upload is newer
- 08:32 PM Revision 9812: inputs/GBIF/table.run: table.tsv*/upload(): renamed to table.tsv.gz/upload() to upload only table.tsv.gz, not table.tsv, in order to save bandwidth
- 08:28 PM Revision 9811: bugfix: lib/sh/sync.sh: also need to --include parent dirs for each --include path
- 08:27 PM Revision 9810: lib/sh/util.sh: added path_parents()
- 08:01 PM Revision 9809: *{.sh,run}: in comments, use ${array[@]} instead of @array for clarity
- 07:54 PM Revision 9808: lib/sh/util.sh: foreach_arg(): moved `local a` to same line as for loop that uses it
- 07:00 PM Revision 9807: bugfix: inputs/GBIF/table.run: table.tsv*/upload(): need to run put in live mode (live=1)
- 06:53 PM Revision 9806: lib/sh/util.sh: foreach_arg(): echo_run the cmd at a log_level up so it isn't printed as if it were an external command (log_level 1)
- 06:33 PM Revision 9805: lib/sh/sync.sh: removed `pf upload` debug statement
- 06:32 PM Revision 9804: bugfix: lib/sh/util.sh: set_fds(): localize $i so it doesn't overwrite any previous value
- 06:30 PM Revision 9803: inputs/GBIF/table.run: table.tsv/make(): run table.tsv*/upload when the file make is done so that the file is backed up to jupiter
- 06:29 PM Revision 9802: inputs/GBIF/table.run: added table.tsv*/upload()
- 06:27 PM Revision 9801: lib/sh/local.sh: added sync_upload(), sync_download() with $sync_local_dir, $sync_remote_url config vars
- 06:26 PM Revision 9800: added lib/sh/sync.sh with upload(), download()
- 06:25 PM Revision 9799: lib/sh/util.sh: added foreach_arg()
- 06:06 PM Revision 9798: bugfix: lib/sh/util.sh: need to use `declare -p` instead of ${var+isset} because ${var+isset} returns not set for empty arrays
- 06:05 PM Revision 9797: lib/sh/util.sh: added echo_vars() stub
- 05:43 PM Revision 9796: lib/sh/util.sh: added echo_run() stub
- 05:39 PM Revision 9795: lib/sh/util.sh: set_fds(): don't run (or echo) exec if no redirections are being made
- 05:37 PM Revision 9794: bugfix: lib/sh/util.sh: added missing stub for indent alias (used by echo_func alias, which is a stub). without the stub, /usr/bin/indent would be used instead on Mac.
- 04:01 PM Revision 9793: bugfix: lib/sh/local.sh: root_rel_path(): added echo_func
- 04:01 PM Revision 9792: bugfix: lib/sh/local.sh: root_rel_path(): use canon_rel_path instead of rel_path because $1 may be absolute rather than relative to the current dir, so $root_dir needs to be absoluted (which requires $1 to be absoluted as well)
- 03:56 PM Revision 9791: lib/sh/util.sh: support custom $base_dir, which will be run through realpath() to match $path ($PWD, which was used before, did not need to be realpath'd because it was already absolute)
- 03:34 PM Revision 9790: lib/sh/util.sh: moved echo_func alias to stub because it must be embedded in its expanded alias form to work properly
- 03:29 PM Revision 9789: lib/sh/util.sh: declare echo_func as a stub before it's defined, so that functions can use it even if they are defined before it (and its logging functionality will be enabled as soon as it's defined)
- 03:26 PM Revision 9788: lib/sh/util.sh: rel_path(): don't log++ it, and instead only log++ applicable calls of it or its callers. this allows non-internal calls of rel_path() to be logged at the usual log_level.
- 01:00 PM Revision 9787: lib/sh/local.sh: added root_rel_path()
- 12:39 PM Revision 9786: lib/Firefox_bookmarks.reformat.csv: unescape HTML in page's description, such as links to more info. this is necessary to properly render the persistent shells link in the screen > scrollback folder description.
- 12:37 PM Revision 9785: bin/repl: added unescape_html() filter function, which can be specified as the replacement string
- 12:35 PM Revision 9784: bin/repl: support Unicode characters in the matched portion of the string
- 11:52 AM Revision 9783: web/links/index.htm: updated to Firefox bookmarks. `screen`: scrollback: added link to our wiki page on persistent shells, which are a better way to support reconnecting.
- 11:36 AM Revision 9782: web/links/index.htm: updated to Firefox bookmarks. added bookmarks about the `screen` command, especially how to access the scrollback. resorted several folders alphabetically.
- 11:13 AM Revision 9781: inputs/GBIF/raw_occurrence_record_plants/run: table.tsv/make(): documented how to view the runtime when using `screen` (press Ctrl-A [ , use up-arrow, and then press Esc to leave copy mode)
- 11:12 AM Revision 9780: inputs/GBIF/raw_occurrence_record_plants/run: herbaria_filter/make(): use new ih_herbarium table instead of the herbaria_filter.ih.csv_ file directly
Also available in: Atom