Project

General

Profile

Activity

From 05/25/2013 to 06/23/2013

06/23/2013

09:30 PM Revision 10015: lib/sh/sync.sh: upload(): documented that each --include path is relative to the *currdir*, not the root dir of the upload ($local_dir). this feature, although previously unintended, is actually better because the user can change to a subdir of the root dir and specify upload paths relative to the dir they are in. however, when invoking upload() from a script with --include paths specified, this means you need to use an absolute path (e.g. "$(dirname "${BASH_SOURCE[0]}")"/...; or the value that will become $local_dir, which for sync_upload() is $root_dir).
Aaron Marcuse-Kubitza
08:56 PM Revision 10014: backups/.rsync_ignore: replaced with .rsync_filter.upload to allow uploading new backups but not deleting existing backups if they don't exist on the local (rsync-invoking) side; and .rsync_filter.download to avoid downloading backups to the local side. this allows storing older backups just on jupiter, where there is much more disk space. note that this change must be made on the *remote* side (jupiter) for it to be effective, because these are remote-side rules and are only processed by the remote-side rsync instance.
Aaron Marcuse-Kubitza
08:55 PM Revision 10013: lib/sh/sync.sh: upload(): use directional .rsync_filter to supplement .rsync_ignore with all kinds of --filter rules. separate .rsync_filters are needed for the upload (swap=) and download (swap=1) directions because the sender and the receiver are reversed, causing asymmetric rules like protect/hide to change meaning.
Aaron Marcuse-Kubitza
07:48 PM Revision 10012: updated backups/TNRS.backup.md5
Aaron Marcuse-Kubitza
07:48 PM Revision 10011: added backups/TNRS.2013-6-17.backup.md5, TNRS.2013-6-22.backup.md5
Aaron Marcuse-Kubitza
03:58 PM Revision 10010: /README.TXT: Backups: TNRS cache: Back up/Restore: added runtimes (3 min/5.5 min)
Aaron Marcuse-Kubitza
03:52 PM Revision 10009: lib/sh/sync.sh: upload(): usage: documented put's swap=1 flag, which downloads instead of uploads
Aaron Marcuse-Kubitza
03:47 PM Revision 10008: added inputs/GBIF/raw_occurrence_record_plants/.rsync_ignore with filters that have previously needed to be manually added whenever `make inputs/upload` was run
Aaron Marcuse-Kubitza
03:46 PM Revision 10007: added inputs/GBIF/_MySQL/.rsync_ignore with filters from /README.TXT > Maintenance > to synchronize vegbiendev, jupiter, and your local machine. these filters will now be used with bin/sync_upload in addition to the periodic backup commands.
Aaron Marcuse-Kubitza
03:45 PM Revision 10006: added inputs/VegBIEN/TWiki/.rsync_ignore with filters from /README.TXT > Maintenance > to synchronize vegbiendev, jupiter, and your local machine. these filters will now be used with bin/sync_upload in addition to the periodic backup commands.
Aaron Marcuse-Kubitza
03:44 PM Revision 10005: added inputs/.rsync_ignore with filters from inputs/Makefile $(rsyncSrcs). these filters will now be used with bin/sync_upload in addition to `make inputs/upload`.
Aaron Marcuse-Kubitza
03:43 PM Revision 10004: added bin/.rsync_ignore with filters from /README.TXT > Maintenance > to synchronize vegbiendev, jupiter, and your local machine. these filters will now be used with bin/sync_upload in addition to the periodic backup commands.
Aaron Marcuse-Kubitza
03:40 PM Revision 10003: added backups/.rsync_ignore with filters from /README.TXT > Maintenance > to synchronize vegbiendev, jupiter, and your local machine. these filters will now be used with bin/sync_upload in addition to the periodic backup commands.
Aaron Marcuse-Kubitza
03:36 PM Revision 10002: /.rsync_ignore: added *.pyc
Aaron Marcuse-Kubitza
03:36 PM Revision 10001: added /.rsync_ignore with filters from lib/common.Makefile $(rsync). these filters will now be used with bin/sync_upload in addition to `make inputs/upload`.
Aaron Marcuse-Kubitza
03:34 PM Revision 10000: lib/sh/sync.sh: upload(): use --exclude filters from per-dir .rsync_ignore. note that --exclude-from can't be used for this, because it is relative to the currdir, not the rsync root, and therefore also requires the .rsync_ignore to exist rather than using it only if it exists.
Aaron Marcuse-Kubitza

06/22/2013

12:23 AM Revision 9999: bin/tnrs_db: documented total runtime (10 days)
Aaron Marcuse-Kubitza

06/21/2013

11:58 PM Revision 9998: bin/tnrs_db: documented current runtime (162 ms/name)
Aaron Marcuse-Kubitza

06/20/2013

10:58 PM Revision 9997: web/links/index.htm: updated to Firefox bookmarks. sorted NCEAS bookmarks to put homepage and support pages first.
Aaron Marcuse-Kubitza
06:21 PM Revision 9996: /README.TXT: Full database import: To run TNRS, etc. after the main import: clarified that you should only run `export version=<version>` if the import is named something *other than* public (i.e. it has not yet replaced the previous public schema)
Aaron Marcuse-Kubitza
06:14 PM Revision 9995: /README.TXT: Full database import: To run TNRS: removed `by_col=1` because by-column mode is not applicable to running TNRS. it is, however, needed when running import_scrub (i.e. `make inputs/<datasrc>/reimport_scrub by_col=1`).
Aaron Marcuse-Kubitza
06:10 PM Revision 9994: inputs/.TNRS/schema.sql: tnrs: vegbiendev update steps: added `make backups/TNRS.backup-remake` to back up TNRS before making changes to it. this provides a more recent restore point than the last import in case the changes mess things up. (however, the last import's backup is usually sufficient unless TNRS has been run since then.)
Aaron Marcuse-Kubitza
05:53 PM Revision 9993: inputs/.TNRS/schema.sql: tnrs_populate_fields(): added VACUUM ANALYZE and runtime (50 s)
Aaron Marcuse-Kubitza
05:42 PM Revision 9992: inputs/.TNRS/schema.sql: tnrs_populate_fields(): updated runtime (16 min)
Aaron Marcuse-Kubitza
05:09 PM Revision 9991: schemas/VegBIEN/taxonomy/higherPlantGroup.xlsx.src.txt: added Brad's comment that there are some holes in the Embryophyte subclasses list, and we need to validate it
Aaron Marcuse-Kubitza
04:49 PM Revision 9990: inputs/.TNRS/schema.sql: tnrs: documented that when changing this table's schema, you must also make the same changes on vegbiendev. included sample util.set_col_types() call with runtime (4 min).
Aaron Marcuse-Kubitza
03:58 PM Revision 9989: inputs/.TNRS/schema.sql: tnrs_populate_fields(): updated runtime (16 min)
Aaron Marcuse-Kubitza
03:32 PM Revision 9988: bugfix: inputs/.TNRS/schema.sql: tnrs_populate_fields(): need to schema-qualify invoked functions
Aaron Marcuse-Kubitza
03:29 PM Revision 9987: bugfix: inputs/.TNRS/schema.sql: tnrs_populate_fields(): Is_homonym: use the *_is_homonym flag for whichever of genus or family (in that order) is NOT NULL, rather than horizontal-ORing potentially NULL values together
Aaron Marcuse-Kubitza
03:22 PM Revision 9986: bugfix: inputs/.TNRS/schema.sql: family_is_homonym(), genus_is_homonym(): need to return NULL instead of false when input family/genus is NULL. EXISTS() does not support this, so STRICT is used to provide this functionality automatically.
Aaron Marcuse-Kubitza
03:19 PM Revision 9985: inputs/.TNRS/schema.sql: added family_is_homonym(), genus_is_homonym() and use them in tnrs_populate_fields()
Aaron Marcuse-Kubitza
03:15 PM Revision 9984: inputs/.TNRS/schema.sql: score_ok(): changed to IMMUTABLE and STRICT
Aaron Marcuse-Kubitza
03:14 PM Revision 9983: inputs/.TNRS/schema.sql: tnrs_populate_fields(): updated runtime (16 min)
Aaron Marcuse-Kubitza
02:41 PM Revision 9982: inputs/.TNRS/schema.sql: tnrs_populate_fields(): never_homonym: use Author_score threshold to exclude matches that are too fuzzy to confirm the presence of a plant name author
Aaron Marcuse-Kubitza
02:38 PM Revision 9981: bugfix: inputs/.TNRS/schema.sql: tnrs_populate_fields(): *_is_homonym: also need to check that there was no Author_matched (i.e. that it could be a homonym). Is_homonym: use new never_homonym var.
Aaron Marcuse-Kubitza
02:18 PM Revision 9980: inputs/.TNRS/schema.sql: tnrs_populate_fields(): updated runtime (18 min)
Aaron Marcuse-Kubitza
02:17 PM Revision 9979: inputs/import.stats.xls: Updated import times
Aaron Marcuse-Kubitza
02:07 PM Revision 9978: planning/workflow/bien3_architecture.pptx: stage II: removed Step prefix before stage #, which the other slides don't have
Aaron Marcuse-Kubitza
01:57 PM Revision 9977: added planning/workflow/bien3_architecture/stages.png
Aaron Marcuse-Kubitza
01:36 PM Revision 9976: added planning/workflow/bien3_architecture/stage_*.png
Aaron Marcuse-Kubitza
12:54 PM Revision 9975: added planning/workflow/bien3_architecture.pptx
Aaron Marcuse-Kubitza
08:20 AM Revision 9974: inputs/.TNRS/schema.sql: tnrs_populate_fields(): when changing this function: UPDATE statement: include TNRS schema since it may not be in the search_path
Aaron Marcuse-Kubitza
08:14 AM Revision 9973: inputs/.TNRS/schema.sql: tnrs_populate_fields(): Is_plant: also consider homonyms using new family_is_homonym, genus_is_homonym (see wiki.vegpath.org/Result_filtering#taxon_is_plant)
Aaron Marcuse-Kubitza
08:03 AM Revision 9972: inputs/.TNRS/schema.sql: tnrs: added Is_homonym derived col (uses IRMNG.family_homonym_epithet, genus_homonym_epithet)
Aaron Marcuse-Kubitza
07:54 AM Revision 9971: schemas/vegbien.sql: re-ran `make schemas/public/reinstall; make schemas/remake` cycle, which apparently changed sort order of statements
Aaron Marcuse-Kubitza
07:11 AM Revision 9970: /README.TXT: Full database import: disk space check: updated minimum (to 300GB) for new import schema size. note that most of the space (166GB) is indexes, and even of the 87GB of data, only 20GB is from GBIF and 15GB from FIA (so most of it is duplication).
Aaron Marcuse-Kubitza
07:07 AM Revision 9969: added inputs/IRMNG/*_homonym_epithet/map.csv, etc. (created by */run)
Aaron Marcuse-Kubitza
07:01 AM Revision 9968: bugfix: inputs/input.Makefile: `%/install %/header.csv: %/create.sql`: in noclobber mode, mark %/header.csv as .PRECIOUS so the existing file won't be deleted if the table already exists (causing an error exit)
Aaron Marcuse-Kubitza
06:54 AM Revision 9967: bugfix: lib/runscripts/table.run: remake_VegBIEN_mappings(): run yes using piped_cmd() so the SIGPIPE doesn't cause an errexit
Aaron Marcuse-Kubitza
06:45 AM Revision 9966: added inputs/IRMNG/{genus_homonym_epithet,family_homonym_epithet}/run, which inherit from ../table.run so that load_data() (which runs create.sql) is invoked
Aaron Marcuse-Kubitza
06:44 AM Revision 9965: added inputs/IRMNG/species_homonyms/new_terms.csv
Aaron Marcuse-Kubitza
06:43 AM Revision 9964: bugfix: added no-op inputs/IRMNG/Source/run so inputs/IRMNG/run would have something to invoke for it
Aaron Marcuse-Kubitza
06:35 AM Revision 9963: inputs/IRMNG/run: use lib/runscripts/datasrc_dir.run, which now provides import() and $subdirs
Aaron Marcuse-Kubitza
06:34 AM Revision 9962: lib/runscripts/datasrc_dir.run: extend import.run and provide an import() implementation that runs all the runscripts for import_order.txt subdirs
Aaron Marcuse-Kubitza
06:20 AM Revision 9961: lib/csvs.py: sniff(): support single-column spreadsheets by defaulting to the Excel dialect when the delimiter can't be determined
Aaron Marcuse-Kubitza
06:09 AM Revision 9960: inputs/IRMNG/: added family_homonym_epithet, genus_homonym_epithet lookup tables, which use util.all_same() to filter out internal Plantae homonyms
Aaron Marcuse-Kubitza
05:44 AM Revision 9959: schemas/util.sql: added all_same() aggregate
Aaron Marcuse-Kubitza
05:31 AM Revision 9958: schemas/util.sql: added not_empty(anyarray)
Aaron Marcuse-Kubitza
12:14 AM Revision 9957: schemas/util.sql: added not_null() (usable as an aggregate's FINALFUNC)
Aaron Marcuse-Kubitza
12:13 AM Revision 9956: schemas/util.sql: added not_null() (usable as an aggregate's FINALFUNC)
Aaron Marcuse-Kubitza

06/19/2013

10:18 PM Revision 9955: bugfix: inputs/IRMNG/import_order.txt: need to specify order so that Source is first
Aaron Marcuse-Kubitza
10:16 PM Revision 9954: bugfix: inputs/IRMNG/*/map.csv: remapped Authority to scientificNameAuthorship instead of authors (now data_authors <VegCore.vegpath.org?data_authors> for clarity)
Aaron Marcuse-Kubitza
09:59 PM Revision 9953: inputs/IRMNG/map.csv: updated to scrubbed output names from */map.csv (/map.csv does not currently get scrubbed)
Aaron Marcuse-Kubitza
09:51 PM Revision 9952: bugfix: inputs/IRMNG/species_homonyms/header.csv, map.csv: reset input columns to DSV (delim-separated values) header. they had gotten changed to the output names in running map.csv with remake=1, causing it to be remade from the (renamed) staging tables.
Aaron Marcuse-Kubitza
08:54 PM Revision 9951: inputs/input.Makefile: $(_svnFilesGlob): added *Makefile
Aaron Marcuse-Kubitza
08:51 PM Revision 9950: /README.TXT: `make inputs/{upload,download}`: first run with test=1 to see what the diffs will be
Aaron Marcuse-Kubitza
08:47 PM Revision 9949: added inputs/IRMNG/, including runscripts to download the names. this is now the 2nd datasource after GBIF to use runscripts, and the 3rd after FIA/GBIF to use new-style import.
Aaron Marcuse-Kubitza
08:45 PM Revision 9948: inputs/input.Makefile: $(_svnFilesGlob): added *run (runscripts)
Aaron Marcuse-Kubitza
08:24 PM Revision 9947: lib/runscripts/table.run: import(): also run remake_VegBIEN_mappings() to accept the test output. this function was previously unused, but was left in for future use when lib/import.sh was translated to lib/runscripts/table.run (it was used in its import.sh form in inputs/FIA/occurrence_all/import).
Aaron Marcuse-Kubitza
08:21 PM Revision 9946: bugfix: lib/runscripts/table.run: remake_VegBIEN_mappings(): need to change to $top_dir before running `rm header.csv map.csv`
Aaron Marcuse-Kubitza
08:18 PM Revision 9945: lib/sh/util.sh: added in_top_dir()
Aaron Marcuse-Kubitza
08:00 PM Revision 9944: lib/runscripts/table.run: remake_VegBIEN_mappings(): only remake header.csv, map.csv if this target is being run directly, to avoid needing to remake them every time. for tables that are views, this instead requires them to be explicitly remade when the view columns change.
Aaron Marcuse-Kubitza
07:07 PM Revision 9943: bugfix: lib/runscripts/subdir.run: subdir_make(): only remake if $remake has been explicitly propagated to subdir_make() by using self_make
Aaron Marcuse-Kubitza
06:51 PM Revision 9942: lib/sh/make.sh: added deferred_check_target_exists alias and use it in check_fake_target_exists
Aaron Marcuse-Kubitza
06:41 PM Revision 9941: added lib/sh/web.sh with curl wrapper
Aaron Marcuse-Kubitza
06:41 PM Revision 9940: lib/sh/make.sh: added check_wildcard_target_exists alias
Aaron Marcuse-Kubitza
06:37 PM Revision 9939: lib/sh/util.sh: added wildcard1 alias
Aaron Marcuse-Kubitza
06:37 PM Revision 9938: lib/sh/util.sh: added echo1()
Aaron Marcuse-Kubitza
06:32 PM Revision 9937: lib/runscripts/table.run: load_data(): first make sure schema is installed
Aaron Marcuse-Kubitza
06:31 PM Revision 9936: lib/runscripts/table.run: added datasrc_make_install()
Aaron Marcuse-Kubitza
06:31 PM Revision 9935: table_make_install(): take $install_log as an overridable kw param to support install logs in different locations
Aaron Marcuse-Kubitza
05:55 PM Revision 9934: lib/runscripts/table.run: load_data(): split noclobber functionality into separate table_make_install() function, which can be used by other install-related targets
Aaron Marcuse-Kubitza
03:27 PM Revision 9933: added schemas/VegBIEN/taxonomy/higherPlantGroup.xlsx.src.txt with Brad's description of how the names were chosen
Aaron Marcuse-Kubitza
02:37 PM Revision 9932: added schemas/VegBIEN/taxonomy/higherPlantGroup.xlsx
Aaron Marcuse-Kubitza
01:50 PM Revision 9931: schemas/VegBIEN/planning/taxonomy/: moved non-VegBIEN-specific resources to planning/resources/taxonomy/. this includes Brad's all-important Nomenclature_excerpt.ppt with the Latin taxonomic hierarchy suffixes on slide 5.
Aaron Marcuse-Kubitza
11:02 AM Revision 9930: bugfix: schemas/vegbien.sql: taxon_trait_view: use the TNRS-scrubbed name from ScrubbedTaxon when available
Aaron Marcuse-Kubitza
10:36 AM Revision 9929: schemas/vegbien.sql: split geoscrub_input_view's new-row-only filtering into separate view geoscrub_input_new, so that the full geoscrub_input rows are still available. the reduction in geoscrub_input from eliminating the already-scrubbed rows was only 280,000 (5076500 - 4799173) out of a possible 1.7 million (1707970), so it makes sense to just run geoscrubbing on the full input. (the lower-than-expected reduction is most likely due to rows from pre-refresh data being present in the original geoscrub_output table, which have been replaced by different, post-refresh input rows.)
Aaron Marcuse-Kubitza
10:18 AM Revision 9928: added exports/_archive/
Aaron Marcuse-Kubitza
10:17 AM Revision 9927: mappings/VegCore-VegBIEN.csv: genus->taxonlabel.taxonomicname: use new _filter_genus() (see r9882)
Aaron Marcuse-Kubitza
10:15 AM Revision 9926: backups/TNRS.backup.md5: updated
Aaron Marcuse-Kubitza
09:43 AM Revision 9925: bin/make_analytical_db: use new mk_table() instead of TRUNCATE/INSERT
Aaron Marcuse-Kubitza
09:41 AM Revision 9924: bin/make_analytical_db: added mk_table() and use it in mk_analytical_table()
Aaron Marcuse-Kubitza
09:30 AM Revision 9923: schemas/vegbien.sql: higher_plant_group_nodes: ferns and allies: added Lycopodiophyta node, as requested by Brad in the conference call (wiki.vegpath.org/2013-06-13_conference_call)
Aaron Marcuse-Kubitza
09:30 AM Revision 9922: schemas/vegbien.sql: geoscrub_input_view: exclude rows that have already been geoscrubbed, by anti-joining on geoscrub_output
Aaron Marcuse-Kubitza
09:11 AM Revision 9921: inputs/.geoscrub/geoscrub_output/postprocess.sql: set decimallatitude, decimallongitude types to double precision to facilitate joining with other double precision values
Aaron Marcuse-Kubitza
09:02 AM Revision 9920: inputs/.geoscrub/geoscrub_output/postprocess.sql: coords index: added rest of input columns so this can be used to check the existence of a result by input. added runtime (55 s). use idempotent create_if_not_exists().
Aaron Marcuse-Kubitza
08:17 AM Revision 9919: schemas/vegbien.sql: higher_plant_group_nodes: ferns and allies: added Lycopodiophyta node, as requested by Brad in the conference call (wiki.vegpath.org/2013-06-13_conference_call)
Aaron Marcuse-Kubitza
08:08 AM Revision 9918: bugfix: schemas/vegbien.sql: higher_plant_group_nodes: removed ferns and allies nodes Anthocerotophyta, Marchantiophyta, Bryophyta, which were incorrectly said to be part of this clade in the BIEN2 analytical DB overview (/planning/workflow/validation/BIEN2_Analytical_DB_overview.docx > p. 13 bottom > last ¶). see http://wiki.vegpath.org/2013-06-13_conference_call#fix-higher_plant_group_nodes-mapping .
Aaron Marcuse-Kubitza
07:58 AM Revision 9917: bugfix: /Makefile: postgres-Linux: phpPgAdmin: added steps to configure it for Apache 2.4
Aaron Marcuse-Kubitza

06/18/2013

07:49 PM Revision 9916: /run: geoscrub_input/make(): documented runtime (40 s)
Aaron Marcuse-Kubitza
06:22 PM Revision 9915: bin/make_analytical_db: added `/run export_` to make the geoscrub_input CSV export
Aaron Marcuse-Kubitza
06:21 PM Revision 9914: inputs/.TNRS/schema.sql: tnrs_populate_fields(): removed no longer needed casts of *_score to double precision
Aaron Marcuse-Kubitza
06:06 PM Revision 9913: inputs/.TNRS/schema.sql: tnrs: *_score: changed type to double precision because these fields are always floats. this also avoids the need to manually cast them to double precision each time they are used.
Aaron Marcuse-Kubitza
05:55 PM Revision 9912: lib/tnrs.py: HTTP requests: rewrapped lines
Aaron Marcuse-Kubitza
05:53 PM Revision 9911: lib/tnrs.py: updated HTTP requests to match current web app
Aaron Marcuse-Kubitza
05:51 PM Revision 9910: bugfix: lib/tnrs.py: download_request_template: changed dirty to true (to match the current web app), which is apparently needed to apply the source_sorting setting to the downloaded TSV in addition to the GUI results
Aaron Marcuse-Kubitza
05:29 PM Revision 9909: lib/tnrs.py: retrieval_request_template: turned source_sorting back off, because it causes any match from the first source to always be used, even if it has a lower match score than the match from the other source. (Brad confirms that this should be off.) I think we had this on originally to ensure that only Tropicos results were used when available, rather than USDA when it was a better match. *** note that due to a bug in the web app, this change will not actually be effective, because the source_sorting option is only applied to the GUI results, not the downloaded TSV. ***
Aaron Marcuse-Kubitza
04:27 PM Revision 9908: inputs/.TNRS/schema.sql: tnrs: Name_number: changed type to integer so it would sort numerically
Aaron Marcuse-Kubitza
04:24 PM Revision 9907: inputs/.TNRS/schema.sql: added pkey on Time_submitted, Name_number
Aaron Marcuse-Kubitza
04:21 PM Revision 9906: inputs/.TNRS/schema.sql: changed Name_submitted pkey to a unique constraint to allow adding a pkey on Time_submitted, Name_number instead
Aaron Marcuse-Kubitza
04:14 PM Revision 9905: inputs/.TNRS/schema.sql: Time_submitted, Name_number: added NOT NULL constraints so that they can be used in a unique constraint
Aaron Marcuse-Kubitza
02:21 PM Revision 9904: lib/tnrs.py: submission_request_template: include GCC in addition to Tropicos, because it provides more synonyms than Tropicos for Asteraceae, and the accepted names still match the Tropicos backbone (https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/2013-06-13_conference_call#include-GCC-when-running-TNRS)
Aaron Marcuse-Kubitza

06/17/2013

08:18 PM Revision 9903: inputs/.TNRS/tnrs/tnrs.make: removed no longer needed end time, now that the total runtime is printed
Aaron Marcuse-Kubitza
08:17 PM Revision 9902: inputs/.TNRS/tnrs/tnrs.make: print the total runtime using `time`
Aaron Marcuse-Kubitza
08:14 PM Revision 9901: inputs/.TNRS/tnrs/tnrs.make: include the end time in addition to the start time so that the total runtime can be calculated
Aaron Marcuse-Kubitza
07:59 PM Revision 9900: lib/sh/util.sh: command-specific alternate stdin/stdout/stderr: choice of 40/41/42: added mnemonic that 4 looks like A for Alternate
Aaron Marcuse-Kubitza

06/14/2013

09:26 AM Task #636 (New): support TNRS-matching names in "", insted of encoding them and letting TNRS return no match
* names like @"Acer rubrum"@ (_with_ quotes) currently get encoded as @ !quo Acer rubrum !quo @, which TNRS can't par... Aaron Marcuse-Kubitza
07:42 AM Revision 9899: bugfix: /README.TXT: Full database import: added step to remove any leftover TNRS lockfile. usually, the PID in it would not exist, but sometimes it now refers to a different, active process which blocks tnrs.make.
Aaron Marcuse-Kubitza

06/13/2013

01:00 PM Revision 9898: schemas/vegbien.sql: allow public_ to view lookup tables (cultivated_family_locations, higher_plant_group_nodes)
Aaron Marcuse-Kubitza
08:13 AM Revision 9897: added backups/TNRS.backup.md5, vegbien.r9459.backup.md5
Aaron Marcuse-Kubitza

06/12/2013

02:45 PM Revision 9896: bugfix: lib/sh/local.sh: sync_upload(): need to use --no-group to prevent the group from being reset to aaronmk upon download from jupiter (which uses group aaronmk instead of bien). use ./fix_perms to set the group of all files to bien. also use --no-owner in case running as root.
Aaron Marcuse-Kubitza
02:12 PM Revision 9895: lib/sh/sync.sh: removed sync_download(). use swap=1 sync_upload() instead.
Aaron Marcuse-Kubitza
02:11 PM Revision 9894: lib/sh/sync.sh: removed download(). use swap=1 upload, or swap=1 upload_caller, instead. this avoids having separate upload()/download() pairs for every caller of upload(), because you can instead just set swap=1.
Aaron Marcuse-Kubitza
01:50 PM Revision 9893: bugfix: lib/sh/sync.sh: upload(): don't kw_params $swap because this unexports it, preventing put from seeing it. instead, use echo_vars to print it.
Aaron Marcuse-Kubitza
01:35 PM Revision 9892: added bin/sync_upload, a wrapper around sync_upload()
Aaron Marcuse-Kubitza
01:23 PM Revision 9891: added bin/sync_upload, a wrapper around sync_upload()
Aaron Marcuse-Kubitza
01:00 PM Revision 9890: bugfix: lib/sh/sync.sh: upload(): only add `--exclude="**"` if there are --includes. this enables running upload() without paths to upload all files.
Aaron Marcuse-Kubitza
12:56 PM Revision 9889: lib/sh/sync.sh: upload(): support passing -- options to put, which will not be run through the path->--include processing
Aaron Marcuse-Kubitza
12:55 PM Revision 9888: bugfix: lib/sh/sync.sh: upload(): added missing `local args=()` initializer
Aaron Marcuse-Kubitza
12:18 PM Revision 9887: /README.TXT: Full database import: On local machine: added step to do steps under Maintenance > "to synchronize vegbiendev, jupiter, and your local machine", which is needed in addition to `make inputs/upload` since that doesn't handle overwrites or deletions
Aaron Marcuse-Kubitza
12:10 PM Revision 9886: /README.TXT: Maintenance: to synchronize vegbiendev, jupiter, and your local machine: added warning that you should pay careful attention to all files that will be deleted or overwritten (as the three machines are often out of sync)
Aaron Marcuse-Kubitza
11:26 AM Revision 9885: added inputs/GBIF/_MySQL/GBIFPortalDB-2013-02-20.data.0.preamble.sql
Aaron Marcuse-Kubitza
11:17 AM Revision 9884: /README.TXT: Full database import: make inputs/{upload,download}: run them first with `test=1` to see what the changes will be
Aaron Marcuse-Kubitza
11:12 AM Revision 9883: /README.TXT: Full database import: `svn up`: use --force to avoid errors about existing files
Aaron Marcuse-Kubitza
10:49 AM Revision 9882: mappings/VegCore-VegBIEN.csv: genus->taxonlabel.taxonomicname: filter out genera that contain numbers (using new _filter_genus()), which break TNRS and prevent it from matching any other parts of the name. later, these genera can instead be moved to the end of the name, where TNRS will correctly match them as Unmatched_terms.
Aaron Marcuse-Kubitza
10:48 AM Revision 9881: bugfix: inputs/VegBIEN/: added _no_import to disable import for this folder, since this is actually just an entry in web/datasources/ with VegPath redirection links, rather than an input to the import process. this fixes "schema "VegBIEN" does not exist" errors generated in `make test`.
Aaron Marcuse-Kubitza
10:45 AM Revision 9880: inputs/input.Makefile: $(dontImport): also support putting a _no_import file at the top level in the datasource to exclude the entire datasource
Aaron Marcuse-Kubitza
10:30 AM Revision 9879: bugfix: lib/sh/local.sh: removed make() override, which is no longer needed now that its operations are performed by verbosity_compat(), and which caused errors by setting $verbosity to the invalid value ""
Aaron Marcuse-Kubitza
10:28 AM Revision 9878: bugfix: lib/sh/util.sh: verbosity_compat(): always use default verbosity (`unset verbosity`) when verbosity == 1, regardless of whether the caller has set $verbosity to the special value "", because $verbosity is supposed to be an integer field and "" is not supported by most functions that use $verbosity. in cases where a util.sh script is invoked, it will set $verbosity back to the default value 1, so this will function as before for util.sh scripts and fix $verbosity for scripts that use a different verbosity system.
Aaron Marcuse-Kubitza
10:05 AM Revision 9877: added inputs/GBIF/raw_occurrence_record_plants/table.tsv.md5
Aaron Marcuse-Kubitza
09:51 AM Revision 9876: inputs/GBIF/raw_occurrence_record_plants/test.xml.ref: regenerated. updated for new staging table input columns, which are now the same as the output columns.
Aaron Marcuse-Kubitza
09:41 AM Revision 9875: bugfix: inputs/input.Makefile: %/VegBIEN.csv: use header from map.csv instead of the new columns, so that source.shortname is set to GBIF instead of VegCore
Aaron Marcuse-Kubitza
09:24 AM Revision 9874: inputs/input.Makefile: %/VegBIEN.csv: when a runscript is available, instead map the *output* columns of map.csv to VegBIEN, because the columns have been renamed in the staging table
Aaron Marcuse-Kubitza
08:32 AM Revision 9873: inputs/GBIF/raw_occurrence_record_plants/VegBIEN.csv: regenerated, which adds row_num input col
Aaron Marcuse-Kubitza
08:16 AM Revision 9872: lib/sh/util.sh: echo_func(): check can_log at beginning of function, so that the resource-intensive func_loc (which calls `readlink -f`) does not need to be called if echo_cmd would not log anything at the current verbosity
Aaron Marcuse-Kubitza
08:02 AM Revision 9871: lib/sh/util.sh: echo_func(): removed no longer used $minor flag. use `clog++... echo_func` instead.
Aaron Marcuse-Kubitza
07:25 AM Revision 9870: lib/sh/util.sh: verbosity_compat(): don't make $verbosity a local var of this function, because then the changes will not be visible to the caller
Aaron Marcuse-Kubitza
07:12 AM Revision 9869: bugfix: bin/make: use verbosity_compat because some make-invoked commands (e.g. bin/map) don't support verbosity=""
Aaron Marcuse-Kubitza
07:10 AM Revision 9868: lib/sh/util.sh: command(): command__exec(): use verbosity_compat to support commands that don't support verbosity=""
Aaron Marcuse-Kubitza
07:09 AM Revision 9867: lib/sh/util.sh: added verbosity_compat(), for use with commands that don't support verbosity=""
Aaron Marcuse-Kubitza
06:47 AM Revision 9866: bugfix: lib/sh/local.sh: make(): when invoking overridden func, need make__make_sh
Aaron Marcuse-Kubitza
06:46 AM Revision 9865: bugfix: lib/sh/util.sh: self, self_sys aliases: need to remove any func_override suffix __* from the FUNCNAME
Aaron Marcuse-Kubitza
06:35 AM Revision 9864: bugfix: inputs/GBIF/import_order.txt, run: updated raw_occurrence_record/ to raw_occurrence_record_plants/
Aaron Marcuse-Kubitza
06:32 AM Revision 9863: inputs/FIA/occurrence_all/test.xml.ref: update inserted row count
Aaron Marcuse-Kubitza
06:27 AM Revision 9862: bugfix: bin/make: include local.sh so that its default verbosity-setting make() override will be used
Aaron Marcuse-Kubitza
06:26 AM Revision 9861: lib/sh/local.sh: added make() override, which uses the default verbosity (i.e. verbosity="") when verbosity == 1. scripts that use lib/sql.py (which uses $verbosity) have different default verbosities, and this default should not be overriden by an env var, unless a higher verbosity has been set.
Aaron Marcuse-Kubitza
06:24 AM Revision 9860: lib/sh/local.sh: added missing include of make.sh, used by root_make()
Aaron Marcuse-Kubitza
05:57 AM Revision 9859: schemas/vegbien.sql: added _filter_genus()
Aaron Marcuse-Kubitza
04:47 AM Revision 9858: inputs/GBIF/raw_occurrence_record_plants/run: import() runtime: specified that this does not include table.tsv.gz/make()
Aaron Marcuse-Kubitza
04:07 AM Revision 9857: inputs/GBIF/raw_occurrence_record_plants/postprocess.sql: Remove institutions that we have direct data for: # duplicates: added revision #
Aaron Marcuse-Kubitza
04:07 AM Revision 9856: inputs/GBIF/raw_occurrence_record_plants/postprocess.sql: Remove institutions that we have direct data for: documented that there are 4.5 million duplicates (59,998,354 rows before - 55,417,646 rows after = 4,580,708)
Aaron Marcuse-Kubitza
03:49 AM Revision 9855: inputs/GBIF/raw_occurrence_record_plants/postprocess.sql: Remove institutions that we have direct data for: added rerun time (~0 thanks to index, so no problem doing the DELETE each time postprocess.sql is run)
Aaron Marcuse-Kubitza
03:25 AM Revision 9854: *{.sh,run}: use simpler .rel() instead of `. "$(dirname "${BASH_SOURCE[0]}")"/...` for relative includes
Aaron Marcuse-Kubitza
03:24 AM Revision 9853: lib/sh/util.sh: added .rel()
Aaron Marcuse-Kubitza
02:52 AM Revision 9852: schemas/VegCore/VegCore.pg.sql: regenerated from installed schema. a Linux bug that caused constraints to be output in reverse order has now been fixed, as of 12.04.2 (broken in 12.04.1).
Aaron Marcuse-Kubitza
02:48 AM Revision 9851: bugfix: inputs/GBIF/_MySQL/MySQL_schema, MySQL_data: sed: put {} commands on their own line to work on Mac
Aaron Marcuse-Kubitza
02:11 AM Revision 9850: lib/sh/util.sh: .(): put echo_func on its own line for clarity
Aaron Marcuse-Kubitza
02:10 AM Revision 9849: lib/sh/util.sh: .(): added echo_func (2 log_levels up because it's internal)
Aaron Marcuse-Kubitza
02:06 AM Revision 9848: schemas/VegCore/mk_derived: use new lib/sh/local.sh instead of lib/import.sh (a precursor to util.sh, etc. still used by inputs/FIA/)
Aaron Marcuse-Kubitza

06/11/2013

07:50 PM Revision 9847: bugfix: load_data(): verbosity_min: use verbosity_min='' so that csv2db's default verbosity (3) is used, instead of setting the verbosity directly to 3, which caused the log++ logging output from bin/make to be echoed at verbosity 3, creating cluttered output
Aaron Marcuse-Kubitza
07:43 PM Revision 9846: lib/sh/util.sh: verbosity_min(): support value '', which sets verbosity=''
Aaron Marcuse-Kubitza
06:40 PM Revision 9845: bugfix: inputs/GBIF/raw_occurrence_record_plants/postprocess.sql: updated column names to match the renamings in map.csv, which are now performed on the staging table itself
Aaron Marcuse-Kubitza
06:38 PM Revision 9844: lib/sh/util.sh: run_args_cmd(): time the command so that the runtime of the outer runscript target (i.e. the command run from the shell) is printed at the end of the output, like in bin/make
Aaron Marcuse-Kubitza
05:56 PM Revision 9843: bugfix: inputs/input.Makefile: %/install: don't run $(cleanup) if it has already been run by $(import_install_), so that it doesn't run twice
Aaron Marcuse-Kubitza
05:54 PM Revision 9842: inputs/input.Makefile: %/postprocess: don't run postprocess.sql if it is supposed to be run by a runscript, because postprocess.sql may then depend on additional steps the runscript runs before it
Aaron Marcuse-Kubitza
05:25 PM Revision 9841: lib/runscripts/table.run: import(): use self_make on load_data so that the remake status determines whether the table is reinstalled
Aaron Marcuse-Kubitza
05:23 PM Revision 9840: bugfix: lib/runscripts/mysql.table.run: import(): added missing set_make_vars, needed by self_make
Aaron Marcuse-Kubitza
05:21 PM Revision 9839: bugfix: lib/runscripts/table.run: load_data(): need to use $_remake instead of $remake when using set_make_vars
Aaron Marcuse-Kubitza
05:20 PM Revision 9838: lib/runscripts/table.run: added set_make_vars to all make targets so $remake would be propagated appropriately
Aaron Marcuse-Kubitza
05:18 PM Revision 9837: lib/runscripts/table.run: load_data(): also clobber install log if remaking, because the table will be reinstalled
Aaron Marcuse-Kubitza
05:12 PM Revision 9836: lib/runscripts/table.run: load_data(): automatically select noclobber mode depending on whether the install log already exists. this removes the need for a separate load_data_first_run() function.
Aaron Marcuse-Kubitza
04:59 PM Revision 9835: bugfix: lib/runscripts/table.run: load_data(): ignore errors if table already exists
Aaron Marcuse-Kubitza
04:52 PM Revision 9834: lib/runscripts/table.run: load_data(): use noclobber=1 to avoid overwriting the install log when re-running the install target idempotently. load_data_first_run() is now available to preserve the output in the log on the first run.
Aaron Marcuse-Kubitza
04:51 PM Revision 9833: inputs/input.Makefile: Staging tables installation: $(logInstall): don't output to the install log if $noclobber flag is set, to prevent overwriting the log when re-running the install target idempotently
Aaron Marcuse-Kubitza
04:18 PM Revision 9832: bugfix: lib/runscripts/mysql.table.run: import(): move previous versions of table.tsv out of the main dir before loading staging tables, to prevent them from being considered a TSV segment file and prepended to table.tsv
Aaron Marcuse-Kubitza
04:15 PM Revision 9831: lib/sh/util.sh: added mv2dir(), mv_glob which wrap mv
Aaron Marcuse-Kubitza
04:14 PM Revision 9830: lib/sh/util.sh: added mkdir alias which adds -p to prevent errors if the dir already exists
Aaron Marcuse-Kubitza
03:45 PM Revision 9829: lib/sh/util.sh: added wildcard alias, similar to make's $(wildcard) function
Aaron Marcuse-Kubitza
03:29 PM Revision 9828: bugfix: inputs/GBIF/raw_occurrence_record_plants/postprocess.sql: institution_code index: create it idempotently using create_if_not_exists() and an explicit index name, so that a duplicate index doesn't get added each time postprocess.sql is run
Aaron Marcuse-Kubitza
03:27 PM Revision 9827: lib/sh/local.sh: psql(): don't put util in the search_path because psql scripts now add it themselves if they need it, using `SELECT util.search_path_append(util);`
Aaron Marcuse-Kubitza
03:22 PM Revision 9826: inputs/GBIF/raw_occurrence_record_plants/postprocess.sql: add util to the search_path so that postprocess.sql will also work when run by inputs/input.Makefile, which only puts the datasource (GBIF) in the search_path
Aaron Marcuse-Kubitza
03:18 PM Revision 9825: schemas/util.sql: added search_path_append()
Aaron Marcuse-Kubitza
03:12 PM Revision 9824: schemas/util.sql: added eval() to allow running EXECUTE outside of a function (and to echo the command that is run)
Aaron Marcuse-Kubitza
09:04 AM Revision 9823: inputs/GBIF/raw_occurrence_record_plants/run: added import() runtime (5 h)
Aaron Marcuse-Kubitza

06/10/2013

11:58 PM Revision 9822: inputs/GBIF/raw_occurrence_record_plants/run: table.tsv.gz/make() runtime: noted that this excludes the upload time
Aaron Marcuse-Kubitza
11:58 PM Revision 9821: inputs/GBIF/raw_occurrence_record_plants/run: added table.tsv.gz/upload() runtime (15 min)
Aaron Marcuse-Kubitza
11:48 PM Revision 9820: added lib/runscripts/mysql.table.run (general to all MySQL datasources) and use it in inputs/GBIF/table.run
Aaron Marcuse-Kubitza
11:13 PM Revision 9819: inputs/GBIF/raw_occurrence_record_plants/run: table.tsv/make(): to view runtime when using `screen`: keys used to scroll: added Ctrl-B/Ctrl-F for page-at-a-time scrolling (there are a lot of pages of output for the import() target!)
Aaron Marcuse-Kubitza

06/09/2013

09:21 PM Revision 9818: bugfix: inputs/GBIF/table.run: table.tsv.gz/make(): don't run table.tsv.gz/upload in test mode, to avoid clobbering the backup of a full table.tsv with a partial, testing table.tsv
Aaron Marcuse-Kubitza
09:18 PM Revision 9817: lib/sh/db.sh: set test mode when using limited # rows
Aaron Marcuse-Kubitza
09:08 PM Revision 9816: bugfix: inputs/GBIF/table.run: table.tsv.gz/upload(): don't use inplace mode because it leaves a newer mtime when aborted, causing rsync to think that the partial upload is actually newer than the source. note that rsync's --partial-dir mode is just as capable of resuming an aborted upload (it will just use a file in .rsync-tmp instead). inplace mode is primarily designed for fixed-offset files which don't change much between edits, but this is not true for exports (or the gzips of them), which will change the file offsets of most data if even one row or column is added or removed.
Aaron Marcuse-Kubitza
09:01 PM Revision 9815: bugfix: inputs/GBIF/table.run: table.tsv.gz/make(): run table.tsv.gz/upload here instead of in table.tsv/make() because it should not run until table.tsv.gz is finished being made, which is not the case in table.tsv/make() because table.tsv.gz/make is run in the background
Aaron Marcuse-Kubitza
08:59 PM Revision 9814: inputs/GBIF/table.run: table.tsv.gz/upload(): moved before table.tsv.gz/make() so it can be used by it
Aaron Marcuse-Kubitza
08:39 PM Revision 9813: bugfix: inputs/GBIF/table.run: table.tsv.gz/upload(): need overwrite=1 because the mtime of an aborted inplace upload is newer
Aaron Marcuse-Kubitza
08:32 PM Revision 9812: inputs/GBIF/table.run: table.tsv*/upload(): renamed to table.tsv.gz/upload() to upload only table.tsv.gz, not table.tsv, in order to save bandwidth
Aaron Marcuse-Kubitza
08:28 PM Revision 9811: bugfix: lib/sh/sync.sh: also need to --include parent dirs for each --include path
Aaron Marcuse-Kubitza
08:27 PM Revision 9810: lib/sh/util.sh: added path_parents()
Aaron Marcuse-Kubitza
08:01 PM Revision 9809: *{.sh,run}: in comments, use ${array[@]} instead of @array for clarity
Aaron Marcuse-Kubitza
07:54 PM Revision 9808: lib/sh/util.sh: foreach_arg(): moved `local a` to same line as for loop that uses it
Aaron Marcuse-Kubitza
07:00 PM Revision 9807: bugfix: inputs/GBIF/table.run: table.tsv*/upload(): need to run put in live mode (live=1)
Aaron Marcuse-Kubitza
06:53 PM Revision 9806: lib/sh/util.sh: foreach_arg(): echo_run the cmd at a log_level up so it isn't printed as if it were an external command (log_level 1)
Aaron Marcuse-Kubitza
06:33 PM Revision 9805: lib/sh/sync.sh: removed `pf upload` debug statement
Aaron Marcuse-Kubitza
06:32 PM Revision 9804: bugfix: lib/sh/util.sh: set_fds(): localize $i so it doesn't overwrite any previous value
Aaron Marcuse-Kubitza
06:30 PM Revision 9803: inputs/GBIF/table.run: table.tsv/make(): run table.tsv*/upload when the file make is done so that the file is backed up to jupiter
Aaron Marcuse-Kubitza
06:29 PM Revision 9802: inputs/GBIF/table.run: added table.tsv*/upload()
Aaron Marcuse-Kubitza
06:27 PM Revision 9801: lib/sh/local.sh: added sync_upload(), sync_download() with $sync_local_dir, $sync_remote_url config vars
Aaron Marcuse-Kubitza
06:26 PM Revision 9800: added lib/sh/sync.sh with upload(), download()
Aaron Marcuse-Kubitza
06:25 PM Revision 9799: lib/sh/util.sh: added foreach_arg()
Aaron Marcuse-Kubitza
06:06 PM Revision 9798: bugfix: lib/sh/util.sh: need to use `declare -p` instead of ${var+isset} because ${var+isset} returns not set for empty arrays
Aaron Marcuse-Kubitza
06:05 PM Revision 9797: lib/sh/util.sh: added echo_vars() stub
Aaron Marcuse-Kubitza
05:43 PM Revision 9796: lib/sh/util.sh: added echo_run() stub
Aaron Marcuse-Kubitza
05:39 PM Revision 9795: lib/sh/util.sh: set_fds(): don't run (or echo) exec if no redirections are being made
Aaron Marcuse-Kubitza
05:37 PM Revision 9794: bugfix: lib/sh/util.sh: added missing stub for indent alias (used by echo_func alias, which is a stub). without the stub, /usr/bin/indent would be used instead on Mac.
Aaron Marcuse-Kubitza
04:01 PM Revision 9793: bugfix: lib/sh/local.sh: root_rel_path(): added echo_func
Aaron Marcuse-Kubitza
04:01 PM Revision 9792: bugfix: lib/sh/local.sh: root_rel_path(): use canon_rel_path instead of rel_path because $1 may be absolute rather than relative to the current dir, so $root_dir needs to be absoluted (which requires $1 to be absoluted as well)
Aaron Marcuse-Kubitza
03:56 PM Revision 9791: lib/sh/util.sh: support custom $base_dir, which will be run through realpath() to match $path ($PWD, which was used before, did not need to be realpath'd because it was already absolute)
Aaron Marcuse-Kubitza
03:34 PM Revision 9790: lib/sh/util.sh: moved echo_func alias to stub because it must be embedded in its expanded alias form to work properly
Aaron Marcuse-Kubitza
03:29 PM Revision 9789: lib/sh/util.sh: declare echo_func as a stub before it's defined, so that functions can use it even if they are defined before it (and its logging functionality will be enabled as soon as it's defined)
Aaron Marcuse-Kubitza
03:26 PM Revision 9788: lib/sh/util.sh: rel_path(): don't log++ it, and instead only log++ applicable calls of it or its callers. this allows non-internal calls of rel_path() to be logged at the usual log_level.
Aaron Marcuse-Kubitza
01:00 PM Revision 9787: lib/sh/local.sh: added root_rel_path()
Aaron Marcuse-Kubitza
12:39 PM Revision 9786: lib/Firefox_bookmarks.reformat.csv: unescape HTML in page's description, such as links to more info. this is necessary to properly render the persistent shells link in the screen > scrollback folder description.
Aaron Marcuse-Kubitza
12:37 PM Revision 9785: bin/repl: added unescape_html() filter function, which can be specified as the replacement string
Aaron Marcuse-Kubitza
12:35 PM Revision 9784: bin/repl: support Unicode characters in the matched portion of the string
Aaron Marcuse-Kubitza
11:52 AM Revision 9783: web/links/index.htm: updated to Firefox bookmarks. `screen`: scrollback: added link to our wiki page on persistent shells, which are a better way to support reconnecting.
Aaron Marcuse-Kubitza
11:36 AM Revision 9782: web/links/index.htm: updated to Firefox bookmarks. added bookmarks about the `screen` command, especially how to access the scrollback. resorted several folders alphabetically.
Aaron Marcuse-Kubitza
11:13 AM Revision 9781: inputs/GBIF/raw_occurrence_record_plants/run: table.tsv/make(): documented how to view the runtime when using `screen` (press Ctrl-A [ , use up-arrow, and then press Esc to leave copy mode)
Aaron Marcuse-Kubitza
11:12 AM Revision 9780: inputs/GBIF/raw_occurrence_record_plants/run: herbaria_filter/make(): use new ih_herbarium table instead of the herbaria_filter.ih.csv_ file directly
Aaron Marcuse-Kubitza

06/08/2013

12:23 PM Revision 9779: inputs/GBIF/raw_occurrence_record_plants/run: added ih_herbarium/make(), which stores the IH herbaria
Aaron Marcuse-Kubitza
11:50 AM Revision 9778: bugfix: inputs/GBIF/raw_occurrence_record_plants/run: table/make(): also filter out rows with a non-plant family (as described at http://vegpath.org/wiki/2013-06-06_conference_call#GBIF-subsetting-fix-raw_occurrence_record-filter-formula), since some institutions have both animal and plant rows, even though they are in IH or in the 80% list. (note that NULL families are OK.)
Aaron Marcuse-Kubitza
04:12 AM Revision 9777: *{.sh,run}: use mysql instead of mysql_ANSI because mysql is now an alias to mysql_ANSI (since ANSI mode still supports key MySQL features, like `` quotes)
Aaron Marcuse-Kubitza
04:09 AM Revision 9776: inputs/GBIF/raw_occurrence_record_plants/run: table.tsv/make(): documented that incremental output is provided right away with --quick (unbuffered), but takes awhile to become visible in Macfusion sshfs. this can be tested with `while true; do stat inputs/GBIF/raw_occurrence_record_plants/table.tsv; sleep 2; done` running concurrently with `./inputs/GBIF/raw_occurrence_record_plants/run table.tsv/make` on vegbiendev:/home/bien/svn .
Aaron Marcuse-Kubitza
04:00 AM Revision 9775: inputs/GBIF/raw_occurrence_record_plants/run: table.tsv/make(): use new raw_occurrence_record_plants view from table/make()
Aaron Marcuse-Kubitza
03:15 AM Revision 9774: bugfix: inputs/GBIF/raw_occurrence_record_plants/run: table/make(): added make of prerequisites
Aaron Marcuse-Kubitza
03:14 AM Revision 9773: bugfix: inputs/GBIF/raw_occurrence_record_plants/run: table/make(): don't reset $table to plant_fraction_for_herbaria_filter for commands that use $table
Aaron Marcuse-Kubitza
03:10 AM Revision 9772: inputs/GBIF/raw_occurrence_record_plants/run: added table/make(), which makes the filter view
Aaron Marcuse-Kubitza
02:14 AM Revision 9771: inputs/GBIF/raw_occurrence_record/: renamed to raw_occurrence_record_plants because it's actually only the plants in raw_occurrence_record, not all of raw_occurrence_record. also, this will allow us to create a separate raw_occurrence_record_plants view whose name matches the folder and does not collide with the raw_occurrence_record table.
Aaron Marcuse-Kubitza
12:44 AM Revision 9770: inputs/GBIF/raw_occurrence_record/run: herbaria_filter/make(): added runtime, which is ~0 since it just needs to do CSV import and index scans
Aaron Marcuse-Kubitza
12:43 AM Revision 9769: inputs/GBIF/raw_occurrence_record/run: herbaria_filter/make(): time the population of herbaria_filter
Aaron Marcuse-Kubitza

06/07/2013

11:47 PM Revision 9768: inputs/GBIF/raw_occurrence_record/run: plant_fraction/make(): updated runtime. added rows affected count to runtime so if the number of rows it's related to (in this case, institution_code) changes, the runtime can be expected to change accordingly.
Aaron Marcuse-Kubitza

06/06/2013

04:54 PM Revision 9767: inputs/.TNRS/schema.sql: tnrs_populate_fields(): documented runtime (17 min)
Aaron Marcuse-Kubitza
04:49 PM Revision 9766: bugfix: inputs/GBIF/raw_occurrence_record/run: plant_fraction/make(): plant_fraction column: COUNT(boolean) counts *non-NULL* rather than true values (which counter-intuitively includes false, because it's non-NULL), so need to add NULLIF(..., false) around the boolean expression to turn it into a NULL-or-not expression. see http://vegpath.org/wiki/2013-06-06_conference_call#GBIF-subsetting-fix-plant_fraction-SQL-bug .
Aaron Marcuse-Kubitza
04:29 PM Revision 9765: inputs/.TNRS/schema.sql: tnrs_populate_fields(): documented that when changing this function, you must regenerate the derived cols using `UPDATE tnrs SET "Name_submitted" = "Name_submitted"`
Aaron Marcuse-Kubitza
04:21 PM Revision 9764: inputs/.TNRS/schema.sql: tnrs_populate_fields(): Is_plant: must match family as Family_score = 1 (as discussed during conference call vegpath.org/wiki/2013-05-30_conference_call#postprocess-TNRS-results-to-exclude-animals-with-genus-homonyms) instead of as Family_matched IS NOT NULL (as listed in Brad's formula at vegpath.org/wiki/Result_filtering#TNRS-results) because TNRS transforms animal to plant families via fuzzy matching, necessitating a Family_score check to ensure an exact match to a plant family that was not transformed from an animal family
Aaron Marcuse-Kubitza
03:29 PM Revision 9763: inputs/.TNRS/schema.sql: added Is_plant derived field, which is populated using the formula at vegpath.org/wiki/Result_filtering#TNRS-results . note that the homonym filtering is currently excluded until we determine whether we can get direct access to the IRMNG homonyms database (http://www.cmar.csiro.au/datacentre/irmng/homonyms.htm). note also that changes to the TNRS schema cannot be fully tested until any TNRS client bugs are fixed, because the data.sql updater requires a working TNRS client to regenerate the sample data.
Aaron Marcuse-Kubitza
02:42 PM Revision 9762: inputs/.TNRS/schema.sql: updated for current TSV schema: renamed Accepted_species->Accepted_name_species, Accepted_family->Accepted_name_family
Aaron Marcuse-Kubitza
02:27 PM Revision 9761: bugfix: schemas/vegbien.sql: tnrs_input_name: must anti-join against MatchedTaxon rather than ValidMatchedTaxon to ensure that *all* of TNRS.tnrs is excluded from the input names. this prevents duplicates from appearing in the TNRS results, which would break the TSV import into TNRS.tnrs. it also prevents no-match names from being scrubbed repeatedly because they were not properly filtered out of the input names.
Aaron Marcuse-Kubitza
02:17 PM Revision 9760: inputs/.TNRS/schema.sql: fixed whitespace
Aaron Marcuse-Kubitza
02:15 PM Revision 9759: inputs/.TNRS/schema.sql: added MatchedTaxon view, which now just renames the columns but does not filter the results, and use it in ValidMatchedTaxon
Aaron Marcuse-Kubitza
02:11 PM Revision 9758: inputs/.TNRS/schema.sql: MatchedTaxon: renamed to ValidMatchedTaxon since this view actually contains only the names with a valid match
Aaron Marcuse-Kubitza
12:59 PM Revision 9757: bugfix: lib/sql.py: parse_exception(): make_DuplicateKeyException(): handle nested exceptions (which should never be generated, but may be in case of sql.py bugs such as wiki.vegpath.org/To_Do#Fixes > #1) by printing the nested exception and then rethrowing the original exception, so that the original exception does not get lost and still ends up at the end of the program's output, to enable debugging
Aaron Marcuse-Kubitza
11:08 AM Revision 9756: inputs/.TNRS/schema.sql: tnrs: documented that when changing this table's schema, you must regenerate data.sql using `inputs/test_taxonomic_names/test_scrub`
Aaron Marcuse-Kubitza
08:09 AM Revision 9755: inputs/GBIF/raw_occurrence_record/run: table.tsv.gz/make(): documented runtime (35 min)
Aaron Marcuse-Kubitza
07:52 AM Revision 9754: bugfix: schemas/vegbien.sql: analytical_stem_view: speciesBinomialWithMorphospecies: if accepted name not specified, use matched name (matched*) or Name_submitted (concatenatedScientificName), as described at http://wiki.vegpath.org/2013-05-30_conference_call#fix-TNRS-speciesBinomialWithMorphospecies-to-include-alternatives-when-no-accepted-name
Aaron Marcuse-Kubitza
07:24 AM Revision 9753: lib/sh/make.sh: make(): time all invocations of make
Aaron Marcuse-Kubitza
07:14 AM Revision 9752: lib/sh/make.sh: make() at verbosity < 4, hide messages about making included Makefiles: use sed with a range expression (/.../,/.../) to also exclude all log messages between an opening "make ...Makefile" and a closing "make[#]: ...Makefile"
Aaron Marcuse-Kubitza
06:35 AM Revision 9751: lib/sh/util.sh: log+ aliases: added clog++/-- aliases for cmds, which don't include log_local. these are useful when you can't just use "log++" because you need the command following it to be alias-expanded.
Aaron Marcuse-Kubitza
05:52 AM Revision 9750: bugfix: lib/sh/make.sh: make(): use [:char_class:] exprs instead of \X char class abbrs because the \ abbrs are not supported on Linux
Aaron Marcuse-Kubitza
05:33 AM Revision 9749: inputs/GBIF/table.run: table.tsv/make(): remake table.tsv.gz/make() after table.tsv is made
Aaron Marcuse-Kubitza
05:27 AM Revision 9748: inputs/GBIF/table.run: added table.tsv.gz/make()
Aaron Marcuse-Kubitza
05:26 AM Revision 9747: lib/sh/util.sh: added bg_cmd() and helpers log_bg(), log_last_bg()
Aaron Marcuse-Kubitza
05:04 AM Revision 9746: lib/runscripts/subdir.run: auto-append -remake to all targets in remake mode
Aaron Marcuse-Kubitza
04:15 AM Revision 9745: lib/runscripts/subdir.run: subdir_make(): put prepending of "$subdir/" on its own line for clarity
Aaron Marcuse-Kubitza
04:00 AM Revision 9744: bugfix: lib/common.Makefile: gzip/gunzip: in `touch -r`, *$<* needs to be the reference file
Aaron Marcuse-Kubitza
02:36 AM Revision 9743: *{.sh,run}: make() calls: removed no longer applicable silent=1, which is now handled automatically by the log_level mechanism
Aaron Marcuse-Kubitza
02:35 AM Revision 9742: /Makefile: new config target (part of install): install our bin/make in the user's ~/bin dir so that bin/make (with filtering of verbose messages) will always be used instead of standard make (without us needing to change every occurrence of make to bin/make!)
Aaron Marcuse-Kubitza
02:25 AM Revision 9741: bugfix: bin/make: use standard make logging port (1) instead of $log_fd (30) so that the output of bin/make can in turn be filtered by util.sh using the standard cmd_log_fd=1
Aaron Marcuse-Kubitza
02:23 AM Revision 9740: lib/sh/util.sh: command(): print the log_fd in effect in case the user changes it
Aaron Marcuse-Kubitza
02:23 AM Revision 9739: lib/sh/make.sh: make(): log the messages about making included Makefiles at log_level 4 instead of 3, so that verbosity 3 can be used to see the vars being used by the filtering process (i.e. the filtering process must still be enabled at verbosity 3)
Aaron Marcuse-Kubitza
02:00 AM Revision 9738: bin/make: use `readlink -f` on BASH_SOURCE[0] so that this script can also be run via a symlink
Aaron Marcuse-Kubitza
01:57 AM Revision 9737: lib/sh/util.sh: no_PATH_recursion(): also remove $top_symlink_dir_abs
Aaron Marcuse-Kubitza
01:56 AM Revision 9736: lib/sh/util.sh: added $top_symlink_dir_abs
Aaron Marcuse-Kubitza
01:53 AM Revision 9735: lib/sh/util.sh: $top_dir_orig: renamed to $top_symlink_dir since this reflects better what the dir actually represents, and when it differs from $top_dir
Aaron Marcuse-Kubitza
01:33 AM Revision 9734: lib/sh/make.sh: make(): fixed func name in usage
Aaron Marcuse-Kubitza
01:32 AM Revision 9733: bin/make: don't print make cmd by default, so that only `make` output is printed at verbosity 1
Aaron Marcuse-Kubitza
01:31 AM Revision 9732: bin/make: don't reinvoke make() if the make filter has already been set up, as indicated by $is_outermost (instead, invoke make directly using exec)
Aaron Marcuse-Kubitza
01:30 AM Revision 9731: lib/sh/util.sh: added $is_outermost, which stores whether the util.sh env has not yet set up
Aaron Marcuse-Kubitza
01:28 AM Revision 9730: lib/sh/util.sh: echo_redirs_cmd(): added $cmd_name_log_inc kw param to allow echoing the command name at a different log_level than the command output
Aaron Marcuse-Kubitza
12:22 AM Revision 9729: lib/sh/make.sh: make(): log the filter setup operations at log_level 2 so they do not clutter up normal output
Aaron Marcuse-Kubitza
12:13 AM Revision 9728: lib/sh/util.sh: filter_fd: alias-expand filter_cmd
Aaron Marcuse-Kubitza
12:08 AM Revision 9727: added bin/make, which runs make, hiding verbose messages about making included Makefiles. this should be used in preference to plain make, to avoid excessive log messages that prevent the user from seeing the core commands that are being run.
Aaron Marcuse-Kubitza
12:05 AM Revision 9726: lib/sh/util.sh: added no_PATH_recursion(), which allows running a system command of the same name as the script
Aaron Marcuse-Kubitza
12:04 AM Revision 9725: lib/sh/util.sh: added PATH_rm(), which removes components from the PATH
Aaron Marcuse-Kubitza
12:03 AM Revision 9724: lib/sh/util.sh: added $top_dir_abs, $top_dir_orig
Aaron Marcuse-Kubitza

06/05/2013

11:44 PM Revision 9723: bugfix: lib/sh/util.sh: command(): use `builtin command` instead of `exec` so that options like -p (reset PATH) work properly. also, the command builtin it overrides is designed to be used with more than just external commands, and command() should not impose additional limitations.
Aaron Marcuse-Kubitza
10:54 PM Revision 9722: lib/sh/util.sh: added self_sys alias, which uses only system utilities (`command -p`) instead of the current PATH
Aaron Marcuse-Kubitza
10:37 PM Revision 9721: lib/sh/make.sh: make(): at verbosity < 3, hide messages about making included Makefiles. this makes the make output **much** more readable when a Makefile contains an include statement, because there won't be a ton of log messages every time a Makefile is included. this filtering is so useful that it probably makes sense to run make for any of our Makefiles using `lib/runscripts/util.run make ...` instead of plain make. compare e.g. `make inputs/ACAD/Specimen/map.csv` (53 lines of output) and `lib/runscripts/util.run make inputs/ACAD/Specimen/map.csv` (17 lines of output, 1/3 as much).
Aaron Marcuse-Kubitza
10:15 PM Revision 9720: lib/sh/util.sh: log++: before cmd: documented that you need to use "log++" instad of log++ to avoid using the log++ alias, which prepends a log_local call. omitting the quotes is generally not a problem, but when there is another command wrapping the log++, you need the "" to avoid the wrapper applying to log_local's declare call instead.
Aaron Marcuse-Kubitza
10:09 PM Revision 9719: lib/sh/util.sh: log++: before cmd: documented how to use it as `log+ #` when incrementing multiple log_levels at once (this is a better method than `"log++" "log++" ...`)
Aaron Marcuse-Kubitza
10:04 PM Revision 9718: lib/sh/util.sh: log++: before cmd: documented that you need to use it as `"log++" "log++"` when repeating it multiple times
Aaron Marcuse-Kubitza
10:02 PM Revision 9717: lib/sh/util.sh: added filter_fd(), which encapsulates the use of >() process substitution for filtering an fd other than stdout (yes, this is possible without lots of 3>&1 1>&2 2>&3 redirections!). this can be useful e.g. to filter logging output or highlight errors.
Aaron Marcuse-Kubitza
09:49 PM Revision 9716: lib/sh/util.sh: log+(): documented that with a cmd, assignments are applied just to the cmd, so log_local is not needed
Aaron Marcuse-Kubitza
08:59 PM Revision 9715: lib/sh/util.sh: moved pipe_delay() before fd-related functions so it can be used by them
Aaron Marcuse-Kubitza
08:56 PM Revision 9714: lib/sh/util.sh: removed no longer needed load new aliases before echo_stdin(), since pipe_delay() is now a function
Aaron Marcuse-Kubitza
08:48 PM Revision 9713: bugfix: lib/sh/util.sh: set_fds(): add #<>&- before every #<>&# reopen: need to use loop var $i instead of $1 (which would have been used with a while/shift method of iterating over $@)
Aaron Marcuse-Kubitza
12:57 PM Revision 9712: bugfix: lib/sh/util.sh: set_fds(): added workaround for strange bash bug where reopening an fd sometimes first requires explicitly closing it, by adding an <>&- entry for every redirection
Aaron Marcuse-Kubitza
12:53 PM Revision 9711: lib/sh/util.sh: added match_prefix()
Aaron Marcuse-Kubitza
03:28 AM Revision 9710: lib/sh/make.sh: make(): determine --silent status based on the verbosity (<=0) instead of a kw param
Aaron Marcuse-Kubitza
03:11 AM Revision 9709: inputs/GBIF/table.run: table.tsv.md5/make(): only use %-remake if remaking
Aaron Marcuse-Kubitza
03:10 AM Revision 9708: lib/sh/make.sh: make(): removed extra space after --silent
Aaron Marcuse-Kubitza

06/04/2013

11:17 PM Revision 9707: lib/sh/db.sh: moved mysql_root() after the mysql->mysql_ANSI alias (and load new aliases) so that it will also use ANSI mode and support "" identifiers
Aaron Marcuse-Kubitza
11:15 PM Revision 9706: lib/sh/db.sh: mysql: always use ANSI mode, to support "" identifiers. note that `` are *still* supported in this mode, so it also works with SHOW CREATE TABLE output and dumpfiles.
Aaron Marcuse-Kubitza
11:05 PM Revision 9705: bugfix: inputs/GBIF/raw_occurrence_record/run: plant_fraction_for_herbaria_filter/make(): need to make prerequisites first (plant_fraction/make)
Aaron Marcuse-Kubitza
06:35 PM Revision 9704: bugfix: inputs/GBIF/table.run: table.tsv.md5/make(): use %-remake to ensure that the .md5 file is remade, regardless of the .md5 file's mtime relative to table.tsv. you would generally expect table.tsv's new mtime to be newer than the .md5 file's (thus triggering make to run), but if you e.g. ran svn up after making the table.tsv, this might not be the case.
Aaron Marcuse-Kubitza
06:31 PM Revision 9703: /Makefile: moved %/remake, %-remake to lib/common.Makefile because they are generally useful
Aaron Marcuse-Kubitza
06:26 PM Revision 9702: /Makefile: moved %/reinstall to lib/common.Makefile because it is generally useful
Aaron Marcuse-Kubitza
06:07 PM Revision 9701: lib/sh/db.sh: mysql_cmd(): run the command with `time` because in mysql()'s output_data mode, no queries, and therefore no runtimes, are echoed, so the total runtime needs to be echoed separately instead. the total runtime is also useful in general, when many long-running queries are run and you would also like to know the total time (e.g. in make_analytical_db).
Aaron Marcuse-Kubitza
05:54 PM Revision 9700: lib/sh/db.sh: limit(): usage: surrounded query in "" to clarify that it's a string, not a command
Aaron Marcuse-Kubitza

06/01/2013

09:32 PM Revision 9699: inputs/GBIF/table.run: table.tsv/make(): use new set_large_table to prevent table.tsv from being deleted on error for full export runs (while still deleting it on error for sample subset runs)
Aaron Marcuse-Kubitza
09:31 PM Revision 9698: lib/sh/db.sh: added set_large_table alias, used to set to_file's $del flag based on $limit
Aaron Marcuse-Kubitza
09:30 PM Revision 9697: lib/sh/util.sh: added exit2bool()
Aaron Marcuse-Kubitza
09:18 PM Revision 9696: lib/sh/util.sh: int2bool(): renamed to int2exit() because it actually sets a boolean exit status rather than returning a boolean string
Aaron Marcuse-Kubitza
09:07 PM Revision 9695: bugfix: lib/sh/util.sh: to_file(): also kw_params the other kw params if_not_exists, del
Aaron Marcuse-Kubitza
07:02 PM Revision 9694: lib/sh/db.sh: mysql_cmd(): use --quick to avoid buffering entire result (which is slow and memory-intensive for large result sets). this option applies to both mysql() and mysqldump().
Aaron Marcuse-Kubitza
06:35 PM Revision 9693: lib/sh/make.sh: self_make(): documented that it should be preceded by set_make_vars
Aaron Marcuse-Kubitza
06:34 PM Revision 9692: bugfix: inputs/GBIF/_MySQL/GBIFPortalDB-2013-02-20.data.sql.run: ^.preamble.sql/make(): need to run set_make_vars even though the make vars are not used, because set_make_vars sets $_remake, which is needed by self_make
Aaron Marcuse-Kubitza
06:33 PM Revision 9691: lib/sh/make.sh: set_make_vars: echo_vars $_remake to help debugging
Aaron Marcuse-Kubitza
06:27 PM Revision 9690: bugfix: lib/sh/util.sh: to_file(): need to run invoked cmd using redir so that >$stdout redir is applied properly when cmd is a shell function instead of an external command (external commands were already redirected properly because command() calls redir). this fixes a bug in inputs/GBIF/_MySQL/GBIFPortalDB-2013-02-20.data.sql.run ^.preamble.sql/make(), where the generated file would be output to stdout instead of to the file because to_file()'s command was a shell function, and therefore the redirection was not applied. this fix requires redir() to be a separate function from command(), because command() does many things that are only applicable to external commands.
Aaron Marcuse-Kubitza
06:20 PM Revision 9689: lib/sh/util.sh: redir(): override save_e and add `unset redirs` so error handlers are not redirected
Aaron Marcuse-Kubitza
06:15 PM Revision 9688: lib/sh/util.sh: added alias_append(), similar to func_override() for aliases
Aaron Marcuse-Kubitza
06:05 PM Revision 9687: lib/sh/util.sh: redir(): don't do redir actions if redir will be run later (i.e. if command to run starts with `redir` or `command`)
Aaron Marcuse-Kubitza
06:04 PM Revision 9686: lib/sh/util.sh: redir(), command(): log their function calls at the usual log_level (2) instead of one higher, so that they are printed in the call tree to help debugging
Aaron Marcuse-Kubitza
05:39 PM Revision 9685: lib/sh/util.sh: added redir() and use it in command() to perform and echo the redirections
Aaron Marcuse-Kubitza
09:35 AM Revision 9684: lib/sh/util.sh: command(): removed comment that "the following redirections must happen in exactly this order", because there is now only one redirection
Aaron Marcuse-Kubitza
06:01 AM Revision 9683: lib/sh/util.sh: command(): removed 2>&$err_fd, which is no longer needed because $err_fd is now 2 (in general, there is probably not a need for a special $err_fd var, because 2 is already stderr)
Aaron Marcuse-Kubitza
05:52 AM Revision 9682: lib/sh/util.sh: command(): don't set err_fd to global stderr, because this prevents errors from being captured and parsed by callers. it is also not necessary to use a separate port than stderr, because stderr already contains only errors (since logging messages go to their own port). global stderr would still be useful e.g. for displaying input prompts the user when reading from global stdin.
Aaron Marcuse-Kubitza
05:17 AM Revision 9681: bugfix: *run: overriding targets: use new self_make to properly progagate the $remake flag to the overridden target, so that the target itself is not skipped
Aaron Marcuse-Kubitza
05:15 AM Revision 9680: lib/sh/util.sh: to_file(): added del= flag to prevent the file from being auto-removed on error (e.g. to preserve a partial result, which normally would be removed)
Aaron Marcuse-Kubitza
05:12 AM Revision 9679: lib/sh/make.sh: added self_make(), which progagates the $remake flag (normally it is not progagated, because prerequisites should not also be remade)
Aaron Marcuse-Kubitza
03:58 AM Revision 9678: lib/sh/util.sh: usage comments: when there is a descriptive comment on the same line as the usage, prepend it with # (as if it were an end-of-line comment) instead of enclosing it in (), to make it visually obvious that it's a comment and not part of the usage commands
Aaron Marcuse-Kubitza
03:54 AM Revision 9677: lib/sh/util.sh: added all_funcs(), which lists all declared functions
Aaron Marcuse-Kubitza
03:31 AM Revision 9676: bugfix: inputs/GBIF/raw_occurrence_record/run: table.tsv/make(): added check_target_exists so table.tsv would not be overwritten if it already existed
Aaron Marcuse-Kubitza
03:30 AM Revision 9675: bugfix: inputs/GBIF/table.run: table.tsv/make(): force table.tsv.md5 to be remade (using remake=1) because the table.tsv contents will have changed
Aaron Marcuse-Kubitza
03:29 AM Revision 9674: bugfix: lib/sh/make.sh: set_make_vars: can't use end-of-line comment in alias because it will comment out whatever is after the alias where it's used. can't just put a newline or ; after the end-of-line comment because the alias's lines will be combined onto one line using ; , so end-of-line comments would not be supported.
Aaron Marcuse-Kubitza
03:23 AM Revision 9673: lib/sh/util.sh: added pf(), to print a function declaration for debugging
Aaron Marcuse-Kubitza
02:08 AM Revision 9672: inputs/GBIF/raw_occurrence_record/run: plant_fraction/make(): documented runtime (1 hr)
Aaron Marcuse-Kubitza
01:19 AM Revision 9671: *.url: mailto URLs: use the standard e-mail dotpath syntax e-mail@host?name.date.subject.(attachment) (vegpath.org/wiki/Global_IDs#Resource). populated missing fields (e.g. name, subject) where needed.
Aaron Marcuse-Kubitza
12:12 AM Revision 9670: *.url: mailto URLs: ensured that they are proper URLs with the mailto: protocol
Aaron Marcuse-Kubitza

05/30/2013

08:32 PM Revision 9669: lib/sh/util.sh: to_file(): exc handler that rm's file: unset redirs so it isn't used in the rm cmd
Aaron Marcuse-Kubitza
08:22 PM Revision 9668: inputs/GBIF/raw_occurrence_record/run: herbaria_filter/make(): removed no longer needed explicit clear of $remake, which is now done by make.sh instead
Aaron Marcuse-Kubitza
08:18 PM Revision 9667: lib/sh/make.sh: set_make_vars: don't progagate remake to prerequisites, so that remake=1 only applies to the outermost target rather than forcing every prerequisite to be remade, too
Aaron Marcuse-Kubitza
07:58 PM Revision 9666: lib/sh/make.sh: moved remaking section before set_make_vars so that it can be used in set_make_vars
Aaron Marcuse-Kubitza
07:53 PM Revision 9665: inputs/GBIF/raw_occurrence_record/run: added herbaria_filter/seal()
Aaron Marcuse-Kubitza
07:51 PM Revision 9664: inputs/GBIF/raw_occurrence_record/run: herbaria_filter/make(): changed "from IH" to "contains all of IH" because not all rows are now from IH
Aaron Marcuse-Kubitza
07:49 PM Revision 9663: inputs/GBIF/raw_occurrence_record/run: herbaria_filter/make(): renamed acronym->institution_code to match the column name in raw_occurrence_record rather than in IH
Aaron Marcuse-Kubitza
07:46 PM Revision 9662: inputs/GBIF/raw_occurrence_record/run: removed no longer used herbaria_filter.plant_fraction.csv_/make(). use plant_fraction_for_herbaria_filter view instead.
Aaron Marcuse-Kubitza
07:45 PM Revision 9661: inputs/GBIF/raw_occurrence_record/run: herbaria_filter/make(): use the plant_fraction_for_herbaria_filter view directly instead of first exporting it to a CSV
Aaron Marcuse-Kubitza
07:19 PM Revision 9660: lib/sh/db.sh: mysql_import(): in append mode, use LOAD DATA IGNORE to allow inserting duplicate rows
Aaron Marcuse-Kubitza
07:09 PM Revision 9659: inputs/GBIF/raw_occurrence_record/run: herbaria_filter/make(): if remaking, turn off remake mode after doing this target's rm operations, so that prerequisite targets are not also remade
Aaron Marcuse-Kubitza
06:56 PM Revision 9658: lib/sh/util.sh: to_file(): removed no longer needed separate logging of >$stdout, which is now done by command()
Aaron Marcuse-Kubitza
06:50 PM Revision 9657: lib/sh/util.sh: echo_redirs_cmd(): use $@ in a subshell instead of manipulating the @redirs array directly, because operations on $@ (e.g. $#, $1, shift) are much simpler than the corresponding array operations ( ${#redirs[@]}, ${redirs[0]}, redirs=("${redirs[@]:1}") )
Aaron Marcuse-Kubitza
06:42 PM Revision 9656: lib/sh/util.sh: echo_redirs_cmd(): log each file redir with a separate log() statement, so each line is indented
Aaron Marcuse-Kubitza
06:38 PM Revision 9655: lib/sh/util.sh: added echo_redirs_cmd and use it in command() to print cmd
Aaron Marcuse-Kubitza
06:32 PM Revision 9654: lib/sh/util.sh: command(): print <>file redirects before command, because they introduce it
Aaron Marcuse-Kubitza
06:31 PM Revision 9653: lib/sh/util.sh: added starts_with()
Aaron Marcuse-Kubitza
05:49 PM Revision 9652: lib/sh/util.sh: to_file(): use @redirs to echo and set >$stdout instead of setting it manually, which is possible now that the command() @redirs bug has been fixed
Aaron Marcuse-Kubitza
05:43 PM Revision 9651: bugfix: lib/sh/util.sh: convention of fds to use for command-specific alternate stdin/stdout/stderr: changed to 40/41/42 because 10/11/12 are used by eval (which is used by set_fds()). use of fd 10/11/12 will cause hard-to-find silent bugs because exec will not print an error when these are used. documented why not to use other series of fds for this purpose:
# do NOT use 1x, which are used by eval (which is used by set_fds())
# do NOT use 2x, which are used as global stdin/...
Aaron Marcuse-Kubitza
02:43 PM Revision 9650: lib/sh/local.sh: psql(): use new psql() from db.sh instead of psql_script_vegbien/psql_verbose_vegbien. this requires setting local_pg_database=vegbien to replace vegbien_dest used by psql_*_vegbien.
Aaron Marcuse-Kubitza
02:38 PM Revision 9649: lib/sh/db.sh: psql(): set $PG* connection env vars from our connection vars ($server, $user, etc.). use use_pg to import $database so it can be different from $database for MySQL
Aaron Marcuse-Kubitza
02:31 PM Revision 9648: lib/sh/db.sh: added use_pg alias
Aaron Marcuse-Kubitza
02:31 PM Revision 9647: bugfix: lib/sh/db.sh: psql(): added missing `--set ON_ERROR_STOP=1 --quiet` opts from psql_script_vegbien
Aaron Marcuse-Kubitza
02:12 PM Revision 9646: lib/sh/db.sh: added psql(), which replaces psql_script_vegbien and psql_verbose_vegbien for general connections. it also supports separate command and stdin files, to allow using `\copy from pstdin`, with pstdin pointing to a separate, EOF-terminated CSV file instead of inlined with the command and terminated with the \. escape (which may be contained within the CSV file itself).
Aaron Marcuse-Kubitza
01:05 PM Revision 9645: bugfix: lib/sh/local.sh: psql(): $file can't both be passed as a --file param and be prefixed with the necessary \set schema, etc. commands, so instead include $file when cat-ing stdin
Aaron Marcuse-Kubitza
08:28 AM Revision 9644: added inputs/GBIF/raw_occurrence_record/postprocess.sql, which removes institutions that we have direct data for
Aaron Marcuse-Kubitza
08:18 AM Revision 9643: inputs/GBIF/raw_occurrence_record/run: herbaria_filter/make(): skip table if already exists (unless remaking), like plant_fraction/make()
Aaron Marcuse-Kubitza
08:16 AM Revision 9642: bugfix: lib/sh/db.sh: mysql_import(): need to use direct connection to DB instead of via ssh, because ssh does not tunnel nonstandard fds
Aaron Marcuse-Kubitza
08:15 AM Revision 9641: lib/sh/db.sh: added ssh2local alias
Aaron Marcuse-Kubitza
07:36 AM Revision 9640: inputs/GBIF/raw_occurrence_record/run: herbaria_filter.plant_fraction.csv_/make(): use new plant_fraction_for_herbaria_filter view
Aaron Marcuse-Kubitza
07:13 AM Revision 9639: inputs/GBIF/raw_occurrence_record/run: added plant_fraction_for_herbaria_filter/make(). note that for simplicity, plant_fraction_for_herbaria_filter is a view instead of a table.
Aaron Marcuse-Kubitza
06:50 AM Revision 9638: inputs/GBIF/raw_occurrence_record/run: *.table/*(): renamed to */*() because a target named after a table refers to the table unless it has an explicit file extension
Aaron Marcuse-Kubitza
06:49 AM Revision 9637: inputs/GBIF/raw_occurrence_record/run: plant_fraction.table/*(): renamed to plant_fraction/*() because a target named after a table refers to the table unless it has an explicit file extension
Aaron Marcuse-Kubitza
06:41 AM Revision 9636: lib/sh/db.sh: mysql_seal_table(): also revoke GRANT OPTION, which apparently needs to be done in addition (and in a separate command, unlike when granting GRANT OPTION)
Aaron Marcuse-Kubitza
06:40 AM Revision 9635: lib/sh/db.sh: mysql_seal_table(): REVOKE: ignore errors if REVOKE was already run
Aaron Marcuse-Kubitza
06:39 AM Revision 9634: lib/sh/db.sh: mysql_seal_table(): REVOKE: removed unneeded explicit database since this is automatically set to the current database
Aaron Marcuse-Kubitza
06:19 AM Revision 9633: inputs/GBIF/raw_occurrence_record/run: added plant_fraction.table/seal(), which uses new mysql_seal_table()
Aaron Marcuse-Kubitza
06:19 AM Revision 9632: lib/sh/db.sh: added mysql_seal_table(), which prevents further modifications to a table by a user. this uses new mysql_root().
Aaron Marcuse-Kubitza
06:18 AM Revision 9631: lib/sh/db.sh: added mysql_root(). this version uses just use_root (compare to the mysql_root() override in local.sh).
Aaron Marcuse-Kubitza
06:16 AM Revision 9630: lib/sh/local.sh: database connection vars: connect to vegbiendev via ssh and run commands locally, to allow running commands as root (which can only connect to the database locally). this effectively requires an ssh account on vegbiendev, but any ssh account (including an anonymous one, if we set one up) will do. this causes schemas/VegCore/VegCore.my.sql, VegCore.pg.sql to change, because they are now created by mysqldump running on vegbiendev (Linux) instead of on a Mac.
Aaron Marcuse-Kubitza

05/29/2013

10:35 PM Revision 9629: inputs/GBIF/raw_occurrence_record/run: plant_fraction: added index on plant_fraction for fast extraction of herbaria by fraction threshold
Aaron Marcuse-Kubitza
10:10 PM Revision 9628: inputs/GBIF/raw_occurrence_record/run: tables: set ENGINE to MyISAM and DEFAULT CHARSET to utf8 to match the other GBIF tables. (note that MyISAM is *not* the default, but is needed to avoid row sort order problems and other issues with InnoDB.)
Aaron Marcuse-Kubitza
08:09 PM Revision 9627: inputs/GBIF/raw_occurrence_record/run: plant_fraction.table/make(): in remaking mode, drop the table first
Aaron Marcuse-Kubitza
08:04 PM Revision 9626: inputs/GBIF/raw_occurrence_record/run: plant_fraction.table/make(): only create and populate the table if it doesn't already exist, to avoid clobbering existing data. the noclobber functionality uses new skip_table(), which is the table analog of require_not_exists().
Aaron Marcuse-Kubitza
08:02 PM Revision 9625: lib/runscripts/table.run, table.run: use new db_make.sh
Aaron Marcuse-Kubitza
08:02 PM Revision 9624: added lib/sh/db_make.sh that includes both db.sh and make.sh, and will eventually contain DB-related make commands
Aaron Marcuse-Kubitza
08:00 PM Revision 9623: lib/sh/db.sh: added skip_table(), which prints an already_exists_msg for tables
Aaron Marcuse-Kubitza
07:56 PM Revision 9622: lib/sh/util.sh: already_exists_msg: undid r9621 because the `|| return 0` should actually always be explicitly specified by the caller, to make it clear that the function will be aborted
Aaron Marcuse-Kubitza
07:47 PM Revision 9621: lib/sh/util.sh: already_exists_msg(): added alias for use as an error handler. note that ..._not_exists() functions should continue to use the "already_exists_msg" function instead to preserve the exit status.
Aaron Marcuse-Kubitza
07:40 PM Revision 9620: lib/sh/util.sh: added already_exists_msg() and use it instead of manually generating the die() call
Aaron Marcuse-Kubitza
07:15 PM Revision 9619: schemas/my.cnf: added innodb_file_per_table so each InnoDB table will get its own file. this should also allow databases with InnoDB tables to be manually renamed.
Aaron Marcuse-Kubitza
07:09 PM Revision 9618: added schemas/my.cnf from /etc/mysql/my.cnf
Aaron Marcuse-Kubitza
06:51 PM Revision 9617: schemas/VegCore/VegCore.my.sql, VegCore.pg.sql: synced to VegCore MySQL DB. for some reason, the fkeys are now output in the opposite order from what they were in before.
Aaron Marcuse-Kubitza
05:22 PM Revision 9616: inputs/.TNRS/schema.sql: MatchedTaxon: filter out rows where Max_score was not high enough to use the TNRS result as a match. removed now-duplicated filter for this in AcceptedTaxon.
Aaron Marcuse-Kubitza
05:19 PM Revision 9615: inputs/.TNRS/schema.sql: ScrubbedTaxon: removed extra ; at end of WHERE clause
Aaron Marcuse-Kubitza
03:48 PM Revision 9614: web/links/index.htm: updated to Firefox bookmarks. some broken favicons have also been fixed, by reopening bookmark in Firefox. (this will only update a favicon *if* there is a newer version. to delete a favicon completely, use Firefox's SQLite Manager plugin.)
Aaron Marcuse-Kubitza
10:17 AM Revision 9613: web/index.php: use XHTML DOCTYPE to match what's used by mod_autoindex. this requires some adjustments in spacing for XHTML's slightly different formatting
Aaron Marcuse-Kubitza
10:15 AM Revision 9612: bugfix: web/.htaccess: need to do DirectoryIndex redirects *before* checking for existing file/dir, because a DirectoryIndexed dir is existing but still needs to be redirected to the index.* file
Aaron Marcuse-Kubitza
10:01 AM Revision 9611: web/.htaccess: mod_autoindex: use the main.css stylesheet to match the look-and-feel of index.php
Aaron Marcuse-Kubitza
10:00 AM Revision 9610: web/.htaccess: mod_autoindex: Note that some listed files are not web-accessible: use ' instead of " to avoid \-escaping embedded "
Aaron Marcuse-Kubitza
09:39 AM Revision 9609: web/.htaccess: mod_autoindex: sort by description when provided, to allow setting a custom (non-alphabetical) sort order using AddDescription
Aaron Marcuse-Kubitza
09:37 AM Revision 9608: web/.htaccess: mod_autoindex: added note that some listed files are not web-accessible. they will produce a "Forbidden" error when clicked.
Aaron Marcuse-Kubitza
09:36 AM Revision 9607: bugfix: web/index.php: added space between the full directory index and the preceding content
Aaron Marcuse-Kubitza
09:35 AM Revision 9606: web/index.php: moved the full directory index within the rest of the document body
Aaron Marcuse-Kubitza
08:58 AM Revision 9605: web/index.php: include full directory index, since the URL patterns list is just a subset of the content available through vegpath.org
Aaron Marcuse-Kubitza
08:25 AM Revision 9604: web/.htaccess: added mod_autoindex IndexOptions, in particular FoldersFirst
Aaron Marcuse-Kubitza
05:26 AM Revision 9603: bugfix: web/.htaccess: changed "mod_dir listing"->"mod_autoindex listing" because mod_dir does not actually handle the autogenerated listings
Aaron Marcuse-Kubitza
05:24 AM Revision 9602: bugfix: web/.htaccess: DirectoryIndex: use disabled instead of on because on is actually treated as a filename, and does not invoke mod_autoindex. the DirectoryIndex directive and the mod_dir module actually apply only to manual index files, not to autogenerated dir listings (which are handled by mod_autoindex).
Aaron Marcuse-Kubitza
04:58 AM Revision 9601: web/index.php: removed no longer needed custom alias j.mp/vegpath# for when page reached through vegbiendev.nceas.ucsb.edu, because vegpath.org is a much more reliable domain than the previous path.vg, and a separate way to reach VegPath when path.vg is down is no longer needed
Aaron Marcuse-Kubitza
04:43 AM Revision 9600: web/.htaccess: <dir>/all forces mod_dir listing: use simpler $mod_dir_listing env var instead of query string modification to indicate that an explicit mod_dir listing should be displayed. this causes /all to replace ?index=1 as the way to force a mod_dir listing. note that the %{ENV:...} test needs to use $REDIRECT_mod_dir_listing instead of $mod_dir_listing, because a redirect will occur between the /all rule and the index.* rule, causing all env vars to be prepended with REDIRECT_ .
Aaron Marcuse-Kubitza
03:48 AM Revision 9599: web/.htaccess: <dir>/all forces mod_dir listing, as a simpler syntax than ?index=1
Aaron Marcuse-Kubitza
03:28 AM Revision 9598: web/.htaccess: for dirs, redirect to index.*: allow requesting a mod_dir listing instead with ?index=1
Aaron Marcuse-Kubitza
03:26 AM Revision 9597: web/.htaccess: handle DirectoryIndex redirects in a RewriteRule instead of with `DirectoryIndex index`, so that RewriteConds can be used to configure when index.* is used as the DirectoryIndex instead of a mod_dir listing
Aaron Marcuse-Kubitza
02:30 AM Revision 9596: web/.htaccess: handle DirectoryIndex subrequests when there is no DirectoryIndex: moved comment about -F subrequest after line it applies to
Aaron Marcuse-Kubitza
02:27 AM Revision 9595: inputs/GBIF/_MySQL/run: documented steps to reload GBIF MySQL
Aaron Marcuse-Kubitza
02:19 AM Revision 9594: web/.htaccess: RewriteRules: added standard [discardpath,noescape,qsappend] options where missing (these should be the default, but aren't)
Aaron Marcuse-Kubitza
 

Also available in: Atom