Activity
From 09/25/2014 to 10/24/2014
10/24/2014
- 06:33 PM Revision 14876: bugfix: /README.TXT: To re-run geoscrubbing: don't close subshell because it closes automatically (this would instead exit `screen`)
- 06:31 PM Revision 14875: /README.TXT: To re-run geoscrubbing: updated runtimes
- 05:38 PM Revision 14874: _license/VegBank_schema/applies_to.txt: added exports/sPlot/BIEN-sPLot-Skype-finalNotes.docx
- 05:36 PM Revision 14873: added exports/sPlot/BIEN-sPLot-Skype-finalNotes.docx from Bob
- 04:10 PM Revision 14872: /README.TXT: To re-run geoscrubbing: updated runtimes
- 03:58 PM Revision 14871: bugfix: /README.TXT: To re-run geoscrubbing: "press Enter" and `unset TMOUT` steps are reversed
- 03:50 PM Revision 14870: bugfix: schemas/public_.sql: geoscrub_input_new: need to join on the ARRAY[] of each column, to compare NULLs properly
- 03:32 PM Revision 14869: bugfix: lib/sh/util.sh: become_sudo don't continue running function after self_sudo finishes
- 01:29 PM Revision 14868: /README.TXT: datasource removal: linked to runtimes on the wiki
- 12:33 PM Revision 14867: bugfix: inputs/input.Makefile: %/map.csv: need to save it if errors occur in unmapped_terms.csv, new_terms.csv
- 12:05 PM Revision 14866: fix: inputs/FIA/PLOT/map.csv: added plotArea_ha, requested by Brian McGill
- 12:04 PM Revision 14865: fix: inputs/FIA/PLOT/map.csv: added plotArea_ha, requested by Brian McGill
- 11:05 AM Revision 14864: added backups/old_backups.txt
- 10:49 AM Revision 14863: added backups/TNRS.2014-6-2.backup.md5
10/23/2014
- 05:37 PM Revision 14862: _license/non-open-source/applies_to.txt: clarified that the things in this list are not embargoed, the issue is just that they are not formally open-source
- 03:43 PM Revision 14861: schemas/public_.sql: view_full_occurrence_individual_view: join to taxonoccurrence: can use a USING join for this because plot.** only exports one locationevent_id column
10/20/2014
- 03:33 PM Revision 14860: inputs/Cyrille_traits/trait_observation/postprocess.sql: accessRights index: added vegbiendev runtime
- 03:29 PM Revision 14859: fix: inputs/Cyrille_traits/trait_observation/postprocess.sql: need to remove private data, as clarified by Cyrille
- 03:13 PM Revision 14858: inputs/publishable datasources.xlsx: updated: Cyrille traits: updated with Cyrille's clarifications on publishability
10/19/2014
- 06:27 PM Bug #965 (Resolved): some VegBank plots are missing the plot name
- 03:24 PM Revision 14857: bugfix: lib/sh/sync.sh, db.sh: *_snapshot() commands that run multiple sudo commands: use become_sudo so that the program won't hang at later sudo prompts. for pg_snapshot() with pg_start_backup(), this also prevents the backup from staying open at the end waiting on a sudo prompt, causing excess WAL logs to accrue.
- 03:16 PM Revision 14856: lib/sh/util.sh: become_sudo: documented that this is for funcs that run multiple sudo commands
- 03:15 PM Revision 14855: lib/sh/util.sh: $sudo_cmd_wrapper: use esc() so this does not need to be embedded with [@] array notation
- 03:09 PM Revision 14854: bugfix: lib/runscripts/util.run: $wrap_fn: this *always* needs to contain `is_runscript=1` because $is_runscript is not (and cannot be) exported
- 03:06 PM Revision 14853: bugfix: lib/sh/util.sh: sudo(): when don't need sudo: do need `command` to have effects of invoking external command
- 03:00 PM Revision 14852: fix: lib/sh/db.sh: pg_snapshot(): first need to run pg_stop_backup() in case a backup was not stopped cleanly
- 03:00 PM Revision 14851: fix: lib/sh/db.sh: pg_stop_backup(): ignore benign "a backup is not in progress" error
- 01:19 PM Revision 14850: lib/sh/util.sh: sudo(): only run sudo if needed
- 01:10 PM Revision 14849: lib/sh/util.sh: sudo(): factored PATH-preserving code out into separate wrapper var that can also be used separately, eg. with `sudo -u`
- 01:03 PM Revision 14848: lib/sh/util.sh: sudo(): moved addition of -E (which should always happen) to separate, overridden sudo() function
- 12:51 PM Revision 14847: lib/sh/util.sh: moved -E to is_callable section because it just applies to that
- 11:35 AM Revision 14846: fix: inputs/input.Makefile: $(svnFilesGlob): only svn:ignore *.log in the top-level dir
10/18/2014
- 07:32 PM Revision 14845: added inputs/XAL/Specimen/data.csv (empty template)
- 05:03 PM Revision 14844: fix: inputs/input.Makefile: add!: verify/: also svn:ignore .zip
- 05:03 PM Revision 14843: fix: inputs/input.Makefile: add!: verify/: also svn:ignore .zip
- 04:24 PM Revision 14842: lib/sh/util.sh: added become_sudo alias
- 04:21 PM Revision 14841: lib/sh/util.sh: added $is_root
10/17/2014
- 04:35 PM Bug #965 (Resolved): some VegBank plots are missing the plot name
- h3. problem
this applies to 358 @location@ rows:... - 03:26 PM Bug #945 (New): incorrect synonym being selected in TNRS
- 03:17 PM Bug #945 (Resolved): incorrect synonym being selected in TNRS
- 02:52 PM Revision 14840: inputs/publishable datasources.xlsx: updated: replaced BIEN2 traits with Cyrille traits
- 01:02 PM Revision 14839: exports/view_full_occurrence_CVS.csv.run: documented runtime (2 min for 5000 rows)
- 11:26 AM Bug #950 (Resolved): fix view_full_occurrence_individual_view rows with is_geovalid NULL
10/16/2014
- 01:51 PM Revision 14838: fix: inputs/publishable datasources.xlsx: updated: conditions of use: Canadensys sources: these should actually be marked as no restrictions, in contrast to Brian E's earlier annotations, because they are public domain: the citation is requested, but not legally required
- 01:46 PM Revision 14837: inputs/publishable datasources.xlsx: updated
- 01:40 PM Revision 14836: inputs/publishable datasources.xlsx: updated: conditions of use: indicated which datasources have no restrictions
- 12:47 PM Bug #963 (New): fix slow loading of pages hosted on vegbiendev
- a temporary workaround is to run the following on vegbiendev whenever it's slow:...
- 10:55 AM Revision 14835: web/BIEN3/Redmine/issues/.htaccess: dest: decoded chars that don't need to be escaped, for clarity
- 10:51 AM Revision 14834: bugfix: web/BIEN3/Redmine/issues/.htaccess: need to store dest in separate var because % is treated specially in the replacement str
- 10:49 AM Revision 14833: inputs/CVS/^taxon_observation.**.sample/test.xml.ref: updated
- 10:49 AM Revision 14832: fix: inputs/CVS/plantConcept_/header.csv: regenerated after fixing the postprocess/cleanup ordering bug (r14827), which had caused header.csv to be incorrectly generated after renames in postprocess.sql were applied
- 10:33 AM Revision 14831: web/BIEN3/Redmine/issues/.htaccess: main page: sort by priority
- 10:14 AM Revision 14830: planning/meetings/BIEN conference call availability.xlsx: updated
- 09:38 AM Task #962 (New): walk through loading of datasources from scratch
- 09:28 AM Task #961 (New): test that staging tables can be reloaded from scratch
- h3. bugs
h3(. newWorldCountries mapping array bounds... - 09:28 AM Task #951 (Resolved): test that `make install` is able to fully re-create vegbiendev
10/15/2014
- 05:14 PM Revision 14829: planning/meetings/BIEN conference call availability.xlsx: updated
- 04:08 PM Revision 14828: planning/meetings/BIEN conference call availability.xlsx: updated: added and populated fall 2014 tab
10/14/2014
- 02:50 PM Revision 14827: bugfix: inputs/input.Makefile: postprocess must be run *after* cleanup rather than before because it depends on the cleanup having been performed.
- this bug was not previously detected because this is only a problem when refreshing a datasource to data in the *same...
- 12:36 PM Revision 14826: lib/sql_io.py: cleanup_table(): trim(): documented that this also converts character varying fields to text
- 11:05 AM Revision 14825: bugfix: inputs/input.Makefile: $(dbExports): also need to put data.sql before clean_up.sql, etc. previously, this ordering had to be done by naming clean_up.sql, etc so they would sort after data.sql alphabetically, but it can be confusing to have to remember to do this. this fixes a bug in the CVS refresh where cvs.~.clean_up.sql was being run before data.sql, causing some private columns to have been deleted before the data was imported into the tables, creating a column mismatch error.
- 10:34 AM Revision 14824: /README.TXT: To re-run geoscrubbing: re-create public-schema views that were cascadingly deleted: listed which views these currently are
- 10:26 AM Revision 14823: /README.TXT: To re-run geoscrubbing: updated runtimes
- 10:20 AM Bug #955 (Resolved): fix geovalidation duplicated rows
- 10:16 AM Task #957 (Resolved): add mechanism to customize NULL-equivalent strings used by staging table import
- 10:09 AM Revision 14822: bugfix: lib/sql_io.py: null_strs_str_default: removed "NA" because this is the abbr for a Spanish province (Navarra). this fixes the 2nd bug of #955, geovalidation duplicated rows.
- 10:00 AM Revision 14821: lib/sql_io.py: cleanup_table(): debug-print null_strs
- 09:56 AM Revision 14820: lib/sql_io.py: null_strs: made it customizable from an env var, since the same list of null_strs doesn't work for all datasources (see #957)
- 09:54 AM Revision 14819: inputs/Cyrille_traits/Makefile: set custom $(null_strs) which handles "NA"
- 09:49 AM Revision 14818: inputs/input.Makefile: pass make var $(null_strs) to invoked commands so it can be used by lib/sql_io.py
- 09:39 AM Revision 14817: fix: *Makefile: changed line endings to \n so that `patch` can work with pasted input. use `svn di --extensions --ignore-eol-style` to verify no diff.
- 08:35 AM Revision 14816: lib/sql_io.py: null_strs: made it customizable from an env var, since the same list of null_strs doesn't work for all datasources (see #957)
10/13/2014
- 07:40 AM Revision 14815: bugfix: web/links/index.htm: updated to Firefox bookmarks: Mac: automatic restart on power loss: also need to disable the UPSShutdownAtLevel/UPSShutdownAtMinutesLeft functions because an orderly shutdown prevents the computer from auto-restarting on power loss. Bookmarks Toolbar: reordered links to put those most relevant to VegBIEN users first.
- 07:25 AM Revision 14814: bugfix: config/dev_machine/com.apple.PowerManagement.plist: disable the UPSShutdownAtLevel/UPSShutdownAtMinutesLeft functions because an orderly shutdown prevents the computer from auto-restarting on power loss
- 07:17 AM Revision 14813: bugfix: /README.TXT: Datasource setup: For MS Access databases: also need to comment out `CREATE DATABASE` because pg_dump_limit is only applied to data.sql (due to requirements of the TNRS datasource)
- 06:31 AM Revision 14812: added inputs/CVS/_src/cvs-eep-archive-2014-10-07-correctedCVSData.{data,schema}.sql.ini
- 06:31 AM Revision 14811: bugfix: inputs/CVS/_src/{data,schema}.sql.ini: sourcefilename: this needs to be on the VM's own HD to avoid crashing MSAccess to PostgreSQL. destinationdatabase: added this back since it is fine to leave this blank.
- 06:23 AM Revision 14810: added inputs/CVS/_src/cvs-eep-archive-2013-10-22-VegBIEN.{data,schema}.sql.ini
- 06:17 AM Revision 14809: added inputs/CVS/_src/{data,schema}.sql.ini
- 05:25 AM Revision 14808: added backups/vegbien.r14673.backup.md5
- 04:45 AM Revision 14807: /README.TXT: To re-run geoscrubbing: updated runtimes
10/12/2014
- 03:58 PM Revision 14806: inputs/publishable datasources.xlsx: updated: use white text on dark backgrounds for better visibility, and to create more visual contrast for the unredistributable indicators
- 03:52 PM Revision 14805: inputs/publishable datasources.xlsx: updated: consolidated Brian E's new columns into a single "conditions?" column. fix: "what is needed to publish it": renamed to "conditions of use/remaining tasks". "conditions of use": changed color scheme to match "publishable?" columns.
10/09/2014
10/07/2014
- 04:47 AM Task #957 (Resolved): add mechanism to customize NULL-equivalent strings used by staging table import
- * this is needed to fix issue #955
h3. fix
* see @lib/sql_io.py@ > @null_strs@
# the @null_strs@ need to be ... - 01:59 AM Bug #955 (New): fix geovalidation duplicated rows
- 12:01 AM Revision 14803: bugfix: lib/tnrs.py: encode_map: also need to encode + because TNRS removes it from the morphospecies (vegpath.org/wiki/CVS_validation#Bobs-revised-document > issue #4)
10/06/2014
10/03/2014
- 05:56 PM Revision 14801: config/dev_machine/com.apple.PowerManagement.plist: UPSShutdownAtLevel: decreased to 5% to allow working longer on UPS power
- 05:53 PM Revision 14800: config/dev_machine/com.apple.PowerManagement.plist: UPSShutdownAtMinutesLeft: decreased to 3 min to allow working longer on UPS power
- 05:36 PM Revision 14799: bugfix: inputs/CVS/plot_/postprocess.sql: locality: site_name should come before directions_to_place because it is at a higher level of granularity
- 05:08 PM Revision 14798: /README.TXT: To re-run geoscrubbing: `rm=1 inputs/.geoscrub/run`: should just need to reinstall the geoscrub_output table
- 05:06 PM Revision 14797: schemas/public_.sql: geoscrub_input_view_modify(): add unique constraint on the same columns as geoscrub.geoscrub_output, as a sanity check to assist future debugging
- 05:04 PM Bug #955 (Resolved): fix geovalidation duplicated rows
- 03:23 PM Bug #955 (Resolved): fix geovalidation duplicated rows
- h3. error...
- 04:57 PM Revision 14796: bugfix: schemas/postgresql.conf: need to set extra_float_digits = 3 (the maximum) to enable verbatim export. this should fix issue #955.
- 03:21 PM Task #883 (Rejected): have import scripts regularly check disk space and pause processes if getting close to limit
- our scripts do not trigger this Postgres bug unless they are running buggy queries
- 03:20 PM Task #904 (Resolved): add MySQL public user to allow accessing the normalized VegCore data dictionary
- 12:37 PM Revision 14795: added inputs/CVS/verify/Review of CVS data in BIEN3-RKP2014Sept7.docx from Bob
- 08:18 AM Revision 14794: added config/dev_machine/com.apple.PowerManagement.plist from /Library/Preferences/SystemConfiguration/com.apple.PowerManagement.plist , so that this file can be easily restored when the system spuriously resets settings in it
10/02/2014
10/01/2014
- 09:51 PM Revision 14792: fix: /README.TXT: To re-run geoscrubbing: `rm=1 inputs/.geoscrub/geoscrub_output/geoscrub.csv.run`: use a subshell to avoid closing screen on errexit
- 09:39 PM Revision 14791: fix: /README.TXT: Single datasource refresh: also back up DB after this
- 09:38 PM Revision 14790: /README.TXT: datasource removal: documented bien2_traits runtime
- 09:35 PM Revision 14789: added inputs/bien2_traits/_no_import since bien2_traits has been replaced by Cyrille_traits
- 09:32 PM Revision 14788: /README.TXT: added datasource removal instructions
- 09:22 PM Revision 14787: fix: /README.TXT: Single datasource refresh: place updated extract: generalized datasrc name
09/30/2014
- 07:45 AM Revision 14786: added inputs/Cyrille_traits/
- 07:36 AM Revision 14785: lib/sql_io.py: null_strs: added N/A and NA (this will remove a common abbr for North America, but we don't use the continent, so this is OK)
- 05:04 AM Revision 14784: mappings/VegCore.htm: regenerated from wiki. made verbatimLocality a synonym of locality since they are used to store the same data.
- 03:25 AM Revision 14783: bugfix: schemas/util.sql: col_rename(): also need to run util.name_truncate() on the names used in the "don't try to rename to itself" comparison
- 01:54 AM Revision 14782: fix: schemas/util.sql: col_rename(): don't try to rename a column to itself (this will cause an error)
- 01:22 AM Revision 14781: schemas/util.sql: set_col_names(): use util.col_rename() so that all renames uses this function
- 12:54 AM Revision 14780: schemas/util.sql: added col_rename()
- 12:02 AM Revision 14779: schemas/util.sql: added quote_ident() wrapper
- 12:01 AM Revision 14778: schemas/util.sql: added name_truncate()
09/26/2014
- 10:57 PM Revision 14777: fix: inputs/input.Makefile: $(nonHeaderSrcs): updated to exclude new header.txt
- 10:21 PM Revision 14776: lib/runscripts/table.run: added check_headers()
- 10:21 PM Revision 14775: bugfix: lib/runscripts/table.run: @srcs: missing [@]
- 10:08 PM Revision 14774: lib/runscripts/table.run: added header() and use it in header.txt()
- 09:50 PM Revision 14773: lib/runscripts/table.run: 1st_src(): use a variable for this instead, to avoid needing to run this function each time it's used, and to make @srcs available
- 09:22 PM Revision 14772: lib/sh/util.sh: wildcard.(): clarified that it only removes . .. when at then beginning of the list
- 07:10 PM Revision 14771: fix: schemas/public_.sql: view_full_occurrence_individual_view: materialize time: updated revision
- 06:45 PM Revision 14770: schemas/public_.sql: view_full_occurrence_individual_view: materialize time: updated (2 days)
- 06:32 PM Revision 14769: web/links/index.htm: updated to Firefox bookmarks: added bash shellshock links. LibreOffice: added steps to convert files from the command line.
- 02:31 AM Revision 14768: lib/runscripts/table.run: added header.txt()
- 02:31 AM Revision 14767: lib/runscripts/table.run: added 1st_src()
- 02:30 AM Revision 14766: inputs/input.Makefile: added %/list_srcs
- 02:30 AM Revision 14765: lib/runscripts/subdir.run: subdir_make(): use new $datasrc_dir
- 02:29 AM Revision 14764: lib/runscripts/subdir.run: added $datasrc_dir
- 02:29 AM Revision 14763: lib/sh/make.sh: make(): added support for $output_data mode which uses --silent
- 12:36 AM Revision 14762: fix: lib/sh/util.sh: already_exists_msg(): changed calling convention to avoid it seeming like `return 0` is run if already_exists_msg() throws an error, when in fact already_exists_msg() is just a command that should be run before returning/errexiting
09/25/2014
- 04:28 PM Revision 14761: fix: inputs/input.Makefile: need to escape $ in commands, including inside comments
- 03:28 PM Revision 14760: bugfix: lib/common.Makefile: $(wildcard/): need final pass with $(wildcard) to support inputs without wildcard chars
- 03:15 PM Revision 14759: bugfix: inputs/input.Makefile: `$(call add*,$(svnFiles))` must be invoked externally to clear the $(wildcard) cache before expanding $(svnFiles)
- 02:48 AM Revision 14758: /README.TXT: Installation: added instructions for inputs/download/live prompts
- 02:44 AM Revision 14757: /README.TXT: Installation: added instructions for MySQL installation prompts
- 02:18 AM Revision 14756: fix: /README.TXT: Installation: "Are you sure you want to continue connecting" prompt comes before "aaronmk@jupiter's password" prompt
- 02:13 AM Revision 14755: /Makefile: phppgadmin-Linux: removed no longer needed $(editApacheConfForPhppgadmin), as phpPgAdmin no longer uses this file (/etc/phppgadmin/apache.conf)
- 02:12 AM Revision 14754: /Makefile: phppgadmin-Linux: removed no longer needed $(editPhppgadminApacheConf)
- 01:58 AM Revision 14753: /Makefile: misc-Linux: added htop
- 01:58 AM Revision 14752: /Makefile: misc-Linux: added jedit
- 01:50 AM Revision 14751: /Makefile: postgres-Linux: added pgadmin3
- 01:45 AM Revision 14750: bugfix: /Makefile: $(postgresReload-*) calls: do not prepend $(MAKE) or - because these are multi-command functions
- 01:36 AM Revision 14749: bugfix: /Makefile: $(bien*Password): need to use = instead of := because these files are not available until config/download/live has run
Also available in: Atom