bugfix: inputs/input.Makefile: postprocess must be run after cleanup rather than before because it depends on the cleanup having been performed.

this bug was not previously detected because this is only a problem when refreshing a datasource to data in the same format: this would attempt to run an existing postprocess.sql, out of order, instead of starting with no postprocess.sql as we usually do....

lib/sql_io.py: cleanup_table(): trim(): documented that this also converts character varying fields to text

bugfix: inputs/input.Makefile: $(dbExports): also need to put data.sql before clean_up.sql, etc. previously, this ordering had to be done by naming clean_up.sql, etc so they would sort after data.sql alphabetically, but it can be confusing to have to remember to do this. this fixes a bug in the CVS refresh where cvs.~.clean_up.sql was being run before data.sql, causing some private columns to have been deleted before the data was imported into the tables, creating a column mismatch error.

/README.TXT: To re-run geoscrubbing: re-create public-schema views that were cascadingly deleted: listed which views these currently are

/README.TXT: To re-run geoscrubbing: updated runtimes

bugfix: lib/sql_io.py: null_strs_str_default: removed "NA" because this is the abbr for a Spanish province (Navarra). this fixes the 2nd bug of #955, geovalidation duplicated rows.

lib/sql_io.py: cleanup_table(): debug-print null_strs

lib/sql_io.py: null_strs: made it customizable from an env var, since the same list of null_strs doesn't work for all datasources (see #957)

inputs/Cyrille_traits/Makefile: set custom $(null_strs) which handles "NA"

inputs/input.Makefile: pass make var $(null_strs) to invoked commands so it can be used by lib/sql_io.py

fix: *Makefile: changed line endings to \n so that `patch` can work with pasted input. use `svn di --extensions --ignore-eol-style` to verify no diff.

lib/sql_io.py: null_strs: made it customizable from an env var, since the same list of null_strs doesn't work for all datasources (see #957)

bugfix: web/links/index.htm: updated to Firefox bookmarks: Mac: automatic restart on power loss: also need to disable the UPSShutdownAtLevel/UPSShutdownAtMinutesLeft functions because an orderly shutdown prevents the computer from auto-restarting on power loss. Bookmarks Toolbar: reordered links to put those most relevant to VegBIEN users first.

bugfix: config/dev_machine/com.apple.PowerManagement.plist: disable the UPSShutdownAtLevel/UPSShutdownAtMinutesLeft functions because an orderly shutdown prevents the computer from auto-restarting on power loss

bugfix: /README.TXT: Datasource setup: For MS Access databases: also need to comment out `CREATE DATABASE` because pg_dump_limit is only applied to data.sql (due to requirements of the TNRS datasource)

added inputs/CVS/_src/cvs-eep-archive-2014-10-07-correctedCVSData.{data,schema}.sql.ini

bugfix: inputs/CVS/_src/{data,schema}.sql.ini: sourcefilename: this needs to be on the VM's own HD to avoid crashing MSAccess to PostgreSQL. destinationdatabase: added this back since it is fine to leave this blank.

added inputs/CVS/_src/cvs-eep-archive-2013-10-22-VegBIEN.{data,schema}.sql.ini

added inputs/CVS/_src/{data,schema}.sql.ini

added backups/vegbien.r14673.backup.md5

/README.TXT: To re-run geoscrubbing: updated runtimes

inputs/publishable datasources.xlsx: updated: use white text on dark backgrounds for better visibility, and to create more visual contrast for the unredistributable indicators

inputs/publishable datasources.xlsx: updated: consolidated Brian E's new columns into a single "conditions?" column. fix: "what is needed to publish it": renamed to "conditions of use/remaining tasks". "conditions of use": changed color scheme to match "publishable?" columns.

added inputs/CVS/verify/Review of CVS data in BIEN3-RKP2014Sept7-Revised.docx from Bob

bugfix: lib/tnrs.py: encode_map: also need to encode + because TNRS removes it from the morphospecies (vegpath.org/wiki/CVS_validation#Bobs-revised-document > issue #4)

/README.TXT: To re-run geoscrubbing: also need to `unset TMOUT` because TMOUT causes shell to exit even with background processes

config/dev_machine/com.apple.PowerManagement.plist: UPSShutdownAtLevel: decreased to 5% to allow working longer on UPS power

config/dev_machine/com.apple.PowerManagement.plist: UPSShutdownAtMinutesLeft: decreased to 3 min to allow working longer on UPS power

bugfix: inputs/CVS/plot_/postprocess.sql: locality: site_name should come before directions_to_place because it is at a higher level of granularity

/README.TXT: To re-run geoscrubbing: `rm=1 inputs/.geoscrub/run`: should just need to reinstall the geoscrub_output table

schemas/public_.sql: geoscrub_input_view_modify(): add unique constraint on the same columns as geoscrub.geoscrub_output, as a sanity check to assist future debugging

bugfix: schemas/postgresql.conf: need to set extra_float_digits = 3 (the maximum) to enable verbatim export. this should fix issue #955.

added inputs/CVS/verify/Review of CVS data in BIEN3-RKP2014Sept7.docx from Bob

added config/dev_machine/com.apple.PowerManagement.plist from /Library/Preferences/SystemConfiguration/com.apple.PowerManagement.plist , so that this file can be easily restored when the system spuriously resets settings in it

/README.TXT: To re-run geoscrubbing: updated runtimes. note that these are faster on the new hardware.

fix: /README.TXT: To re-run geoscrubbing: `rm=1 inputs/.geoscrub/geoscrub_output/geoscrub.csv.run`: use a subshell to avoid closing screen on errexit

fix: /README.TXT: Single datasource refresh: also back up DB after this

/README.TXT: datasource removal: documented bien2_traits runtime

added inputs/bien2_traits/_no_import since bien2_traits has been replaced by Cyrille_traits

/README.TXT: added datasource removal instructions

fix: /README.TXT: Single datasource refresh: place updated extract: generalized datasrc name

added inputs/Cyrille_traits/

lib/sql_io.py: null_strs: added N/A and NA (this will remove a common abbr for North America, but we don't use the continent, so this is OK)

mappings/VegCore.htm: regenerated from wiki. made verbatimLocality a synonym of locality since they are used to store the same data.

bugfix: schemas/util.sql: col_rename(): also need to run util.name_truncate() on the names used in the "don't try to rename to itself" comparison

fix: schemas/util.sql: col_rename(): don't try to rename a column to itself (this will cause an error)

schemas/util.sql: set_col_names(): use util.col_rename() so that all renames uses this function

schemas/util.sql: added col_rename()

schemas/util.sql: added quote_ident() wrapper

schemas/util.sql: added name_truncate()

fix: inputs/input.Makefile: $(nonHeaderSrcs): updated to exclude new header.txt

lib/runscripts/table.run: added check_headers()

bugfix: lib/runscripts/table.run: srcs: missing []

lib/runscripts/table.run: added header() and use it in header.txt()

lib/runscripts/table.run: 1st_src(): use a variable for this instead, to avoid needing to run this function each time it's used, and to make @srcs available

lib/sh/util.sh: wildcard.(): clarified that it only removes . .. when at then beginning of the list

fix: schemas/public_.sql: view_full_occurrence_individual_view: materialize time: updated revision

schemas/public_.sql: view_full_occurrence_individual_view: materialize time: updated (2 days)

web/links/index.htm: updated to Firefox bookmarks: added bash shellshock links. LibreOffice: added steps to convert files from the command line.

lib/runscripts/table.run: added header.txt()

lib/runscripts/table.run: added 1st_src()

inputs/input.Makefile: added %/list_srcs

lib/runscripts/subdir.run: subdir_make(): use new $datasrc_dir

lib/runscripts/subdir.run: added $datasrc_dir

lib/sh/make.sh: make(): added support for $output_data mode which uses --silent

fix: lib/sh/util.sh: already_exists_msg(): changed calling convention to avoid it seeming like `return 0` is run if already_exists_msg() throws an error, when in fact already_exists_msg() is just a command that should be run before returning/errexiting

fix: inputs/input.Makefile: need to escape $ in commands, including inside comments

bugfix: lib/common.Makefile: $(wildcard/): need final pass with $(wildcard) to support inputs without wildcard chars

bugfix: inputs/input.Makefile: `$(call add*,$(svnFiles))` must be invoked externally to clear the $(wildcard) cache before expanding $(svnFiles)

/README.TXT: Installation: added instructions for inputs/download/live prompts

/README.TXT: Installation: added instructions for MySQL installation prompts

fix: /README.TXT: Installation: "Are you sure you want to continue connecting" prompt comes before "aaronmk@jupiter's password" prompt

/Makefile: phppgadmin-Linux: removed no longer needed $(editApacheConfForPhppgadmin), as phpPgAdmin no longer uses this file (/etc/phppgadmin/apache.conf)

/Makefile: phppgadmin-Linux: removed no longer needed $(editPhppgadminApacheConf)

/Makefile: misc-Linux: added htop

/Makefile: misc-Linux: added jedit

/Makefile: postgres-Linux: added pgadmin3

bugfix: /Makefile: $(postgresReload-*) calls: do not prepend $(MAKE) or - because these are multi-command functions

bugfix: /Makefile: $(bien*Password): need to use = instead of := because these files are not available until config/download/live has run

fix: /README.TXT: Installation: added instructions for "aaronmk@jupiter's password" prompt

/README.TXT: Installation: added step to open a terminal window

fix: /README.TXT: Installation: added instructions for "Are you sure you want to continue connecting" prompt

inputs/VegBank/run*.log: updated. this adds the function call context in addition to the function location.

bugfix: /Makefile: $(uutils_install): need to exit after printing instructions instead of continuing

fix: /README.TXT: Installation: need instructions for "reload PATH" prompt

fix: /README.TXT: Installation: `apt-get install subversion`: need --yes

bugfix: /Makefile: bin/install: uutils: need to reopen terminal window before can proceed with other installation steps

/Makefile: python-Linux: removed no longer used installation of pp (Parallel Python). we use Python's multiprocessing package instead.

bugfix: /Makefile: install: must be config/download/live, not config/download

README.TXT: bin/sync_upload calls as root: don't need to set $USER because this is now set to the correct value in bin/sync_upload's scripts

bugfix: lib/sh/local.sh: $sync_remote_url: jupiter user is always aaronmk, not the local user

*{.sh,run}: use standard WARNING syntax for warning labels

bugfix: /Makefile: bin/install: also need to install uutils

*{.sh,run}: use standard WARNING syntax for warning labels

fix: lib/sh/util.sh: verbosity_compat(): documented that this should not be run until right before executing an external command, so that it doesn't mess up the logging mechanism

bugfix: bin/make: don't run verbosity_compat until right before executing the external command, so that it doesn't mess up the logging mechanism. this is run automatically by command(), so there is no need to do anything here. note that logging bugs like these can now be troubleshooted much more easily with pst() to narrow down which functions could be causing the problem.

bugfix: lib/sh/util.sh: functions called by pst(): commented out/removed logging calls that would have caused infinite recursion when pst()'ing a logging function

lib/sh/util.sh: echo_func(): get call context before wrapper(s), which is more useful for debugging. this uses skip_stack_frames()'s lookahead=1 mode.

fix: lib/sh/util.sh: skip_stack_frames() callers: updated for new skip_stack_frames rather than get_stack_frame behavior

lib/sh/util.sh: skip_stack_frames(): added lookahead support, which looks at entry after current to deterine whether to skip current. this is useful for skipping wrappers, by looking at the calling function's name.