Activity
From 09/23/2014 to 10/22/2014
10/20/2014
- 03:33 PM Revision 14860: inputs/Cyrille_traits/trait_observation/postprocess.sql: accessRights index: added vegbiendev runtime
- 03:29 PM Revision 14859: fix: inputs/Cyrille_traits/trait_observation/postprocess.sql: need to remove private data, as clarified by Cyrille
- 03:13 PM Revision 14858: inputs/publishable datasources.xlsx: updated: Cyrille traits: updated with Cyrille's clarifications on publishability
10/19/2014
- 06:27 PM Bug #965 (Resolved): some VegBank plots are missing the plot name
- 03:24 PM Revision 14857: bugfix: lib/sh/sync.sh, db.sh: *_snapshot() commands that run multiple sudo commands: use become_sudo so that the program won't hang at later sudo prompts. for pg_snapshot() with pg_start_backup(), this also prevents the backup from staying open at the end waiting on a sudo prompt, causing excess WAL logs to accrue.
- 03:16 PM Revision 14856: lib/sh/util.sh: become_sudo: documented that this is for funcs that run multiple sudo commands
- 03:15 PM Revision 14855: lib/sh/util.sh: $sudo_cmd_wrapper: use esc() so this does not need to be embedded with [@] array notation
- 03:09 PM Revision 14854: bugfix: lib/runscripts/util.run: $wrap_fn: this *always* needs to contain `is_runscript=1` because $is_runscript is not (and cannot be) exported
- 03:06 PM Revision 14853: bugfix: lib/sh/util.sh: sudo(): when don't need sudo: do need `command` to have effects of invoking external command
- 03:00 PM Revision 14852: fix: lib/sh/db.sh: pg_snapshot(): first need to run pg_stop_backup() in case a backup was not stopped cleanly
- 03:00 PM Revision 14851: fix: lib/sh/db.sh: pg_stop_backup(): ignore benign "a backup is not in progress" error
- 01:19 PM Revision 14850: lib/sh/util.sh: sudo(): only run sudo if needed
- 01:10 PM Revision 14849: lib/sh/util.sh: sudo(): factored PATH-preserving code out into separate wrapper var that can also be used separately, eg. with `sudo -u`
- 01:03 PM Revision 14848: lib/sh/util.sh: sudo(): moved addition of -E (which should always happen) to separate, overridden sudo() function
- 12:51 PM Revision 14847: lib/sh/util.sh: moved -E to is_callable section because it just applies to that
- 11:35 AM Revision 14846: fix: inputs/input.Makefile: $(svnFilesGlob): only svn:ignore *.log in the top-level dir
10/18/2014
- 07:32 PM Revision 14845: added inputs/XAL/Specimen/data.csv (empty template)
- 05:03 PM Revision 14844: fix: inputs/input.Makefile: add!: verify/: also svn:ignore .zip
- 05:03 PM Revision 14843: fix: inputs/input.Makefile: add!: verify/: also svn:ignore .zip
- 04:24 PM Revision 14842: lib/sh/util.sh: added become_sudo alias
- 04:21 PM Revision 14841: lib/sh/util.sh: added $is_root
10/17/2014
- 04:35 PM Bug #965 (Resolved): some VegBank plots are missing the plot name
- h3. problem
this applies to 358 @location@ rows:... - 03:26 PM Bug #945 (New): incorrect synonym being selected in TNRS
- 03:17 PM Bug #945 (Resolved): incorrect synonym being selected in TNRS
- 02:52 PM Revision 14840: inputs/publishable datasources.xlsx: updated: replaced BIEN2 traits with Cyrille traits
- 01:02 PM Revision 14839: exports/view_full_occurrence_CVS.csv.run: documented runtime (2 min for 5000 rows)
- 11:26 AM Bug #950 (Resolved): fix view_full_occurrence_individual_view rows with is_geovalid NULL
10/16/2014
- 01:51 PM Revision 14838: fix: inputs/publishable datasources.xlsx: updated: conditions of use: Canadensys sources: these should actually be marked as no restrictions, in contrast to Brian E's earlier annotations, because they are public domain: the citation is requested, but not legally required
- 01:46 PM Revision 14837: inputs/publishable datasources.xlsx: updated
- 01:40 PM Revision 14836: inputs/publishable datasources.xlsx: updated: conditions of use: indicated which datasources have no restrictions
- 12:47 PM Bug #963 (New): fix slow loading of pages hosted on vegbiendev
- a temporary workaround is to run the following on vegbiendev whenever it's slow:...
- 10:55 AM Revision 14835: web/BIEN3/Redmine/issues/.htaccess: dest: decoded chars that don't need to be escaped, for clarity
- 10:51 AM Revision 14834: bugfix: web/BIEN3/Redmine/issues/.htaccess: need to store dest in separate var because % is treated specially in the replacement str
- 10:49 AM Revision 14833: inputs/CVS/^taxon_observation.**.sample/test.xml.ref: updated
- 10:49 AM Revision 14832: fix: inputs/CVS/plantConcept_/header.csv: regenerated after fixing the postprocess/cleanup ordering bug (r14827), which had caused header.csv to be incorrectly generated after renames in postprocess.sql were applied
- 10:33 AM Revision 14831: web/BIEN3/Redmine/issues/.htaccess: main page: sort by priority
- 10:14 AM Revision 14830: planning/meetings/BIEN conference call availability.xlsx: updated
- 09:38 AM Task #962 (New): walk through loading of datasources from scratch
- 09:28 AM Task #961 (New): test that staging tables can be reloaded from scratch
- h3. bugs
h3(. newWorldCountries mapping array bounds... - 09:28 AM Task #951 (Resolved): test that `make install` is able to fully re-create vegbiendev
10/15/2014
- 05:14 PM Revision 14829: planning/meetings/BIEN conference call availability.xlsx: updated
- 04:08 PM Revision 14828: planning/meetings/BIEN conference call availability.xlsx: updated: added and populated fall 2014 tab
10/14/2014
- 02:50 PM Revision 14827: bugfix: inputs/input.Makefile: postprocess must be run *after* cleanup rather than before because it depends on the cleanup having been performed.
- this bug was not previously detected because this is only a problem when refreshing a datasource to data in the *same...
- 12:36 PM Revision 14826: lib/sql_io.py: cleanup_table(): trim(): documented that this also converts character varying fields to text
- 11:05 AM Revision 14825: bugfix: inputs/input.Makefile: $(dbExports): also need to put data.sql before clean_up.sql, etc. previously, this ordering had to be done by naming clean_up.sql, etc so they would sort after data.sql alphabetically, but it can be confusing to have to remember to do this. this fixes a bug in the CVS refresh where cvs.~.clean_up.sql was being run before data.sql, causing some private columns to have been deleted before the data was imported into the tables, creating a column mismatch error.
- 10:34 AM Revision 14824: /README.TXT: To re-run geoscrubbing: re-create public-schema views that were cascadingly deleted: listed which views these currently are
- 10:26 AM Revision 14823: /README.TXT: To re-run geoscrubbing: updated runtimes
- 10:20 AM Bug #955 (Resolved): fix geovalidation duplicated rows
- 10:16 AM Task #957 (Resolved): add mechanism to customize NULL-equivalent strings used by staging table import
- 10:09 AM Revision 14822: bugfix: lib/sql_io.py: null_strs_str_default: removed "NA" because this is the abbr for a Spanish province (Navarra). this fixes the 2nd bug of #955, geovalidation duplicated rows.
- 10:00 AM Revision 14821: lib/sql_io.py: cleanup_table(): debug-print null_strs
- 09:56 AM Revision 14820: lib/sql_io.py: null_strs: made it customizable from an env var, since the same list of null_strs doesn't work for all datasources (see #957)
- 09:54 AM Revision 14819: inputs/Cyrille_traits/Makefile: set custom $(null_strs) which handles "NA"
- 09:49 AM Revision 14818: inputs/input.Makefile: pass make var $(null_strs) to invoked commands so it can be used by lib/sql_io.py
- 09:39 AM Revision 14817: fix: *Makefile: changed line endings to \n so that `patch` can work with pasted input. use `svn di --extensions --ignore-eol-style` to verify no diff.
- 08:35 AM Revision 14816: lib/sql_io.py: null_strs: made it customizable from an env var, since the same list of null_strs doesn't work for all datasources (see #957)
10/13/2014
- 07:40 AM Revision 14815: bugfix: web/links/index.htm: updated to Firefox bookmarks: Mac: automatic restart on power loss: also need to disable the UPSShutdownAtLevel/UPSShutdownAtMinutesLeft functions because an orderly shutdown prevents the computer from auto-restarting on power loss. Bookmarks Toolbar: reordered links to put those most relevant to VegBIEN users first.
- 07:25 AM Revision 14814: bugfix: config/dev_machine/com.apple.PowerManagement.plist: disable the UPSShutdownAtLevel/UPSShutdownAtMinutesLeft functions because an orderly shutdown prevents the computer from auto-restarting on power loss
- 07:17 AM Revision 14813: bugfix: /README.TXT: Datasource setup: For MS Access databases: also need to comment out `CREATE DATABASE` because pg_dump_limit is only applied to data.sql (due to requirements of the TNRS datasource)
- 06:31 AM Revision 14812: added inputs/CVS/_src/cvs-eep-archive-2014-10-07-correctedCVSData.{data,schema}.sql.ini
- 06:31 AM Revision 14811: bugfix: inputs/CVS/_src/{data,schema}.sql.ini: sourcefilename: this needs to be on the VM's own HD to avoid crashing MSAccess to PostgreSQL. destinationdatabase: added this back since it is fine to leave this blank.
- 06:23 AM Revision 14810: added inputs/CVS/_src/cvs-eep-archive-2013-10-22-VegBIEN.{data,schema}.sql.ini
- 06:17 AM Revision 14809: added inputs/CVS/_src/{data,schema}.sql.ini
- 05:25 AM Revision 14808: added backups/vegbien.r14673.backup.md5
- 04:45 AM Revision 14807: /README.TXT: To re-run geoscrubbing: updated runtimes
10/12/2014
- 03:58 PM Revision 14806: inputs/publishable datasources.xlsx: updated: use white text on dark backgrounds for better visibility, and to create more visual contrast for the unredistributable indicators
- 03:52 PM Revision 14805: inputs/publishable datasources.xlsx: updated: consolidated Brian E's new columns into a single "conditions?" column. fix: "what is needed to publish it": renamed to "conditions of use/remaining tasks". "conditions of use": changed color scheme to match "publishable?" columns.
10/09/2014
10/07/2014
- 04:47 AM Task #957 (Resolved): add mechanism to customize NULL-equivalent strings used by staging table import
- * this is needed to fix issue #955
h3. fix
* see @lib/sql_io.py@ > @null_strs@
# the @null_strs@ need to be ... - 01:59 AM Bug #955 (New): fix geovalidation duplicated rows
- 12:01 AM Revision 14803: bugfix: lib/tnrs.py: encode_map: also need to encode + because TNRS removes it from the morphospecies (vegpath.org/wiki/CVS_validation#Bobs-revised-document > issue #4)
10/06/2014
10/03/2014
- 05:56 PM Revision 14801: config/dev_machine/com.apple.PowerManagement.plist: UPSShutdownAtLevel: decreased to 5% to allow working longer on UPS power
- 05:53 PM Revision 14800: config/dev_machine/com.apple.PowerManagement.plist: UPSShutdownAtMinutesLeft: decreased to 3 min to allow working longer on UPS power
- 05:36 PM Revision 14799: bugfix: inputs/CVS/plot_/postprocess.sql: locality: site_name should come before directions_to_place because it is at a higher level of granularity
- 05:08 PM Revision 14798: /README.TXT: To re-run geoscrubbing: `rm=1 inputs/.geoscrub/run`: should just need to reinstall the geoscrub_output table
- 05:06 PM Revision 14797: schemas/public_.sql: geoscrub_input_view_modify(): add unique constraint on the same columns as geoscrub.geoscrub_output, as a sanity check to assist future debugging
- 05:04 PM Bug #955 (Resolved): fix geovalidation duplicated rows
- 03:23 PM Bug #955 (Resolved): fix geovalidation duplicated rows
- h3. error...
- 04:57 PM Revision 14796: bugfix: schemas/postgresql.conf: need to set extra_float_digits = 3 (the maximum) to enable verbatim export. this should fix issue #955.
- 03:21 PM Task #883 (Rejected): have import scripts regularly check disk space and pause processes if getting close to limit
- our scripts do not trigger this Postgres bug unless they are running buggy queries
- 03:20 PM Task #904 (Resolved): add MySQL public user to allow accessing the normalized VegCore data dictionary
- 12:37 PM Revision 14795: added inputs/CVS/verify/Review of CVS data in BIEN3-RKP2014Sept7.docx from Bob
- 08:18 AM Revision 14794: added config/dev_machine/com.apple.PowerManagement.plist from /Library/Preferences/SystemConfiguration/com.apple.PowerManagement.plist , so that this file can be easily restored when the system spuriously resets settings in it
10/02/2014
10/01/2014
- 09:51 PM Revision 14792: fix: /README.TXT: To re-run geoscrubbing: `rm=1 inputs/.geoscrub/geoscrub_output/geoscrub.csv.run`: use a subshell to avoid closing screen on errexit
- 09:39 PM Revision 14791: fix: /README.TXT: Single datasource refresh: also back up DB after this
- 09:38 PM Revision 14790: /README.TXT: datasource removal: documented bien2_traits runtime
- 09:35 PM Revision 14789: added inputs/bien2_traits/_no_import since bien2_traits has been replaced by Cyrille_traits
- 09:32 PM Revision 14788: /README.TXT: added datasource removal instructions
- 09:22 PM Revision 14787: fix: /README.TXT: Single datasource refresh: place updated extract: generalized datasrc name
09/30/2014
- 07:45 AM Revision 14786: added inputs/Cyrille_traits/
- 07:36 AM Revision 14785: lib/sql_io.py: null_strs: added N/A and NA (this will remove a common abbr for North America, but we don't use the continent, so this is OK)
- 05:04 AM Revision 14784: mappings/VegCore.htm: regenerated from wiki. made verbatimLocality a synonym of locality since they are used to store the same data.
- 03:25 AM Revision 14783: bugfix: schemas/util.sql: col_rename(): also need to run util.name_truncate() on the names used in the "don't try to rename to itself" comparison
- 01:54 AM Revision 14782: fix: schemas/util.sql: col_rename(): don't try to rename a column to itself (this will cause an error)
- 01:22 AM Revision 14781: schemas/util.sql: set_col_names(): use util.col_rename() so that all renames uses this function
- 12:54 AM Revision 14780: schemas/util.sql: added col_rename()
- 12:02 AM Revision 14779: schemas/util.sql: added quote_ident() wrapper
- 12:01 AM Revision 14778: schemas/util.sql: added name_truncate()
09/26/2014
- 10:57 PM Revision 14777: fix: inputs/input.Makefile: $(nonHeaderSrcs): updated to exclude new header.txt
- 10:21 PM Revision 14776: lib/runscripts/table.run: added check_headers()
- 10:21 PM Revision 14775: bugfix: lib/runscripts/table.run: @srcs: missing [@]
- 10:08 PM Revision 14774: lib/runscripts/table.run: added header() and use it in header.txt()
- 09:50 PM Revision 14773: lib/runscripts/table.run: 1st_src(): use a variable for this instead, to avoid needing to run this function each time it's used, and to make @srcs available
- 09:22 PM Revision 14772: lib/sh/util.sh: wildcard.(): clarified that it only removes . .. when at then beginning of the list
- 07:10 PM Revision 14771: fix: schemas/public_.sql: view_full_occurrence_individual_view: materialize time: updated revision
- 06:45 PM Revision 14770: schemas/public_.sql: view_full_occurrence_individual_view: materialize time: updated (2 days)
- 06:32 PM Revision 14769: web/links/index.htm: updated to Firefox bookmarks: added bash shellshock links. LibreOffice: added steps to convert files from the command line.
- 02:31 AM Revision 14768: lib/runscripts/table.run: added header.txt()
- 02:31 AM Revision 14767: lib/runscripts/table.run: added 1st_src()
- 02:30 AM Revision 14766: inputs/input.Makefile: added %/list_srcs
- 02:30 AM Revision 14765: lib/runscripts/subdir.run: subdir_make(): use new $datasrc_dir
- 02:29 AM Revision 14764: lib/runscripts/subdir.run: added $datasrc_dir
- 02:29 AM Revision 14763: lib/sh/make.sh: make(): added support for $output_data mode which uses --silent
- 12:36 AM Revision 14762: fix: lib/sh/util.sh: already_exists_msg(): changed calling convention to avoid it seeming like `return 0` is run if already_exists_msg() throws an error, when in fact already_exists_msg() is just a command that should be run before returning/errexiting
09/25/2014
- 04:28 PM Revision 14761: fix: inputs/input.Makefile: need to escape $ in commands, including inside comments
- 03:28 PM Revision 14760: bugfix: lib/common.Makefile: $(wildcard/): need final pass with $(wildcard) to support inputs without wildcard chars
- 03:15 PM Revision 14759: bugfix: inputs/input.Makefile: `$(call add*,$(svnFiles))` must be invoked externally to clear the $(wildcard) cache before expanding $(svnFiles)
- 02:48 AM Revision 14758: /README.TXT: Installation: added instructions for inputs/download/live prompts
- 02:44 AM Revision 14757: /README.TXT: Installation: added instructions for MySQL installation prompts
- 02:18 AM Revision 14756: fix: /README.TXT: Installation: "Are you sure you want to continue connecting" prompt comes before "aaronmk@jupiter's password" prompt
- 02:13 AM Revision 14755: /Makefile: phppgadmin-Linux: removed no longer needed $(editApacheConfForPhppgadmin), as phpPgAdmin no longer uses this file (/etc/phppgadmin/apache.conf)
- 02:12 AM Revision 14754: /Makefile: phppgadmin-Linux: removed no longer needed $(editPhppgadminApacheConf)
- 01:58 AM Revision 14753: /Makefile: misc-Linux: added htop
- 01:58 AM Revision 14752: /Makefile: misc-Linux: added jedit
- 01:50 AM Revision 14751: /Makefile: postgres-Linux: added pgadmin3
- 01:45 AM Revision 14750: bugfix: /Makefile: $(postgresReload-*) calls: do not prepend $(MAKE) or - because these are multi-command functions
- 01:36 AM Revision 14749: bugfix: /Makefile: $(bien*Password): need to use = instead of := because these files are not available until config/download/live has run
09/24/2014
- 11:53 PM Revision 14748: fix: /README.TXT: Installation: added instructions for "aaronmk@jupiter's password" prompt
- 11:49 PM Revision 14747: /README.TXT: Installation: added step to open a terminal window
- 11:48 PM Revision 14746: fix: /README.TXT: Installation: added instructions for "Are you sure you want to continue connecting" prompt
- 11:34 PM Revision 14745: inputs/VegBank/run*.log: updated. this adds the function call context in addition to the function location.
- 11:28 PM Revision 14744: bugfix: /Makefile: $(uutils_install): need to exit after printing instructions instead of continuing
- 11:13 PM Revision 14743: fix: /README.TXT: Installation: need instructions for "reload PATH" prompt
- 11:10 PM Revision 14742: fix: /README.TXT: Installation: `apt-get install subversion`: need --yes
- 10:58 PM Revision 14741: bugfix: /Makefile: bin/install: uutils: need to reopen terminal window before can proceed with other installation steps
- 09:04 PM Revision 14740: /Makefile: python-Linux: removed no longer used installation of pp (Parallel Python). we use Python's multiprocessing package instead.
- 07:57 PM Revision 14739: bugfix: /Makefile: install: must be config/download/live, not config/download
- 07:52 PM Revision 14738: README.TXT: bin/sync_upload calls as root: don't need to set $USER because this is now set to the correct value in bin/sync_upload's scripts
- 07:50 PM Revision 14737: bugfix: lib/sh/local.sh: $sync_remote_url: jupiter user is always aaronmk, not the local user
- 07:41 PM Revision 14736: *{.sh,run}: use standard **WARNING** syntax for warning labels
- 07:32 PM Revision 14735: bugfix: /Makefile: bin/install: also need to install uutils
- 07:26 PM Revision 14734: *{.sh,run}: use standard **WARNING** syntax for warning labels
- 07:25 PM Revision 14733: fix: lib/sh/util.sh: verbosity_compat(): documented that this should not be run until right before executing an external command, so that it doesn't mess up the logging mechanism
- 07:23 PM Revision 14732: bugfix: bin/make: don't run verbosity_compat until right before executing the external command, so that it doesn't mess up the logging mechanism. this is run automatically by command(), so there is no need to do anything here. note that logging bugs like these can now be troubleshooted much more easily with pst() to narrow down which functions could be causing the problem.
- 06:55 PM Revision 14731: bugfix: lib/sh/util.sh: functions called by pst(): commented out/removed logging calls that would have caused infinite recursion when pst()'ing a logging function
- 06:38 PM Revision 14730: lib/sh/util.sh: echo_func(): get call context before wrapper(s), which is more useful for debugging. this uses skip_stack_frames()'s lookahead=1 mode.
- 06:37 PM Revision 14729: fix: lib/sh/util.sh: skip_stack_frames() callers: updated for new skip_stack_frames rather than get_stack_frame behavior
- 06:32 PM Revision 14728: lib/sh/util.sh: skip_stack_frames(): added lookahead support, which looks at entry *after* current to deterine whether to skip current. this is useful for skipping wrappers, by looking at the *calling* function's name.
- 06:30 PM Revision 14727: fix: lib/sh/util.sh: skip_stack_frames(): for new skip_stack_frames rather than get_stack_frame behavior, stack frames must be skipped in the caller to preserve the stack frame pointer
- 06:26 PM Revision 14726: lib/sh/util.sh: get_stack_frame_after(): renamed to skip_stack_frames() for clarity
- 06:25 PM Revision 14725: lib/sh/util.sh: added skip_stack_frame_in_caller, unskip_stack_frame_in_caller
- 06:24 PM Revision 14724: lib/sh/util.sh: added prev_stack_frame
- 07:04 AM Revision 14723: lib/sh/util.sh: echo_func(): use new format_stack_frame, which adds call context information to what was provided by func_loc
- 07:02 AM Revision 14722: fix: lib/sh/util.sh: format_stack_frame(): need to hide canon_rel_path() info using log+
- 07:01 AM Revision 14721: lib/sh/util.sh: added get_stack_frame_after()
- 06:59 AM Revision 14720: lib/sh/util.sh: added matches()
- 06:58 AM Revision 14719: lib/sh/util.sh: next_stack_frame: documented usage
- 04:24 AM Revision 14718: fix: lib/runscripts/util.run: runscript template: all(): moved example commands to target(), where they would more likely be located
- 02:35 AM Revision 14717: lib/sh/util.sh: format_stack_frame(): support including args
- 12:08 AM Revision 14716: lib/sh/util.sh: debugging: added pst() (print_stack_trace)
- 12:05 AM Revision 14715: lib/sh/util.sh: added stack_trace(), print_stack_trace()
- 12:04 AM Revision 14714: lib/sh/util.sh: added format_stack_frame()
- 12:03 AM Revision 14713: lib/sh/util.sh: added get_stack_frame() and helpers
09/23/2014
Also available in: Atom