lib/util.sh: mysql_cmd (): use ${FUNCNAME1} to get the caller's command name instead of requiring the caller to pass the command name explicitly
lib/util.sh: databases: added use_remote alias to use the remote_* connection vars (which uses new import_vars)
lib/util.sh: added import_vars alias
lib/util.sh: mysql_cmd (): added doc comment
lib/util.sh: mysql (): use new mysql_cmd to auto-add connection/login opts when specified
lib/util.sh: added mysql_cmd ()
lib/util.sh: added local_inv alias
lib/util.sh: zip/unzip_newer: support extracting/updating the file even if it has an older mtime by setting $force, using new set_inv to get a var to use to determine whether to include the -u flag
lib/util.sh: added set_inv ()
lib/util.sh: echo_vars (): prefix the declare output with "+ " because it's actually a command, not just an indication of the var's value
lib/util.sh: added sed ()
lib/util.sh: added section labels
web/links/index.htm: removed broken favicons
web/links/index.htm: removed broken favicons. this requires using the SQLite Manager Firefox add-on <https://addons.mozilla.org/en-US/firefox/addon/sqlite-manager/?src=search> to manually remove the favicon entries.
lib/Firefox_bookmarks.reformat.csv: formatting fixes: don't put the src attribute of the favicon image on its own line, because it no longer contains long inline image data
lib/Firefox_bookmarks.reformat.csv: favicon image: remove ICON attributes with inline image data, since these are no longer needed for the favicon images and this avoids cluttering up the svn diff
lib/Firefox_bookmarks.reformat.csv: favicon image: use ICON_URI instead of ICON to fetch the favicon directly from the original site. this allows the favicons to be cached separately from the bookmarks page, so that the inline favicon contents don't need to be re-retrieved every time the bookmarks page is updated. this also avoids cluttering up the svn diff.
web/links/index.htm: updated to Firefox bookmarks. added description of the formats supported by PostgreSQL's COPY command and their escape sequences.
bugfix: schemas/vegbien.sql: geoscrub_input_view: only include places whose country is not NULL, because at least the country is required for geoscrubbing
/README.TXT: Schema changes: Reinstall staging tables: added step to run `make schemas/public/install` after the datasources are installed. note that this must be run after the datasources are installed, because views in public depend on some of the datasources.
/README.TXT: Schema changes: Reinstall staging tables: use `make db` instead of mk_db, which includes the required schemas/install command. some datasources use the util schema in their postprocess scripts, so this must be installed before the staging tables.
bugfix: /Makefile: moved schemas/install from install to db so that it is also run whenever the DB is reinstalled. schemas/install is needed by the staging tables reinstallation performed by reinstall_all.
bugfix: /README.TXT: Schema changes: Reinstall staging tables: also need to run `make mk_db` before running reinstall_all
/README.TXT: Schema changes: Reinstall staging tables: drop the saved previous DB at the end of the operation
/README.TXT: Schema changes: Reinstall staging tables: run all of the local machine steps on vegbiendev, including the vegbien DB rename, to ensure that the existing data is saved in case the reinstall has errors
/README.TXT: Schema changes: Reinstall staging tables: added step to first rename existing vegbien database, so it is not modified in the reinstallation
inputs/FIA/: archived no longer used BIEN2 FIA data from the nimoy geoscrub DB
inputs/FIA/_archive/: moved FIA_COND_unique, Organism into 2011-10-17/ subdir
/README.TXT: Schema changes: Reinstall staging tables: added step to first run this on the local machine to check for any errors, before running it on the live DB, to prevent the live DB from becoming unrestorable in the case of bugs
bugfix: bin/with_all: pipe yes to the command in case it makes any interactive prompts, as in e.g. the TNRS staging tables reinstall
bugfix: inputs/input.Makefile: sql/install: manually specify $no_search_path option to psql_script_vegbien, which is added automatically in $(psqlNoSearchPath) but that uses psql_verbose_vegbien
bugfix: *{.sh,run}: use `local` when setting default values for params to prevent these values from leaking into the calling context, where it may have unexpected effects (e.g. pg_copy_to would permanently set $source, which would then be used on the next invocation of pg_copy_to instead of regenerating $source from the next $table value). this involves using ${var-default} or ${var:-default} to return the value to the local statement instead of setting it directly with ${var=default} or ${var:=default}.
*{.sh,run}: commands that take named arguments: usage messages: changed to use simpler bash syntax for specifying command-specific env vars, which does not require a subshell ("The environment for any simple command or function may be augmented temporarily by prefixing it with parameter assignments" <http://www.gnu.org/software/bash/manual/bash.html#Environment>)
schemas/pg_hba*.conf: removed trailing whitespace
schemas/pg_hba*.conf: allow the bien group to access all databases, including test (on the Mac). note that this does not grant access to the vegbank DB on vegbiendev, because that is owned by postgres.
web/links/index.htm: explicitly separate page's description from author comments where needed
web/links/index.htm: use - instead of --- to separate the page's description from the author comments because it's faster to type and one hyphen is sufficient
lib/Firefox_bookmarks.reformat.csv: labeling page's description: don't match descriptions with a leading blank line as an author comment, because sometimes page descriptions themselves start with a blank line. instead, author comments at the beginning of bookmark descriptions should always start with a " or be preceded by a line containing only hyphens (-). note that it was never reliable to use a blank line for this purpose, because it would be trimmed out upon reimport of the HTML file into Firefox.
web/links/index.htm: updated to Firefox bookmarks
inputs/NVS/*/map.csv: Taxon Growth Form: use new _lowercase() to avoid needing to manually map each value that is already valid but just needs to be lowercased
schemas/util.sql: added _lowercase()
added inputs/CTFS/ERD/ VegPath location
added inputs/MO/_src/_README.TXT with warning that the refresh provided to us is missing several columns from the original extract: CollectionCode, CatalogNumber, IndividualCount
schemas/VegCore/VegCore.ERD.mwb: reduced MaxColumnsDisplayed to 10 to prevent wide tables from taking up too much space in the ERD. this was a big problem in the VegBIEN ERD, which we are hoping to prevent in the VegCore ERD as first-class columns get added to the tables. note that it's very important to ensure that the first-class columns are sorted in importance order, so that the 10 most important are visible in the ERD.
schemas/VegCore/VegCore.ERD.mwb: renamed specimen_holder_institution to specimenholder_institution because specimenholder is one word
lib/sql_io.py: put_table(): Calling wrapper function: adding pkey or index on the resulting table: don't display warning if a pkey can't be added, because this is actually a legitimate situation when the called function is set-returning and can return multiple rows for one input. having this as a warning results in spurious warnings in the automated tests (which look confusingly like ignored errors because Python warnings include debugging context information). e.g. `make inputs/Madidi/IndividualObservation/test.by_col.xml` causes this error in the sourcelist->sourcename splitting step (which of course can produce multiple specimenholder institutions)....
bugfix: lib/sql.py: parse_exception(): typed_name_re (used by MissingCastException, etc.): don't require "" around the value, because function names in "function does not exist" exceptions are (now?) output without them. this exception did not occur in the last import, but does occur in the automated testing and when running the associated query in pgAdmin (in both Linux and Mac OS X).
bugfix: /Makefile: apache-Linux: use the standard apt-get syntax for Makefiles (`-sudo apt-get --yes install`) instead of just `apt-get install`
/Makefile: added Apache 2.4 installation
lib/util.sh: added join ()
web/.twiki/data/Main/TWikiPreferences.txt: updated with current settings
put web/.twiki/data/Main/TWikiPreferences.txt under version control. note that this file (Main/TWikiPreferences.txt) is the correct file to change the settings in, not TWiki/TWikiPreferences.txt.
put web/.twiki/data/TWiki/TWikiPreferences.txt under version control
inputs/VegBIEN/TWiki/data/TWiki/TinyMCEPlugin.txt: twikibuttons_formats: added blockquote format
inputs/VegBIEN/TWiki/data/TWiki/TinyMCEPlugin.txt: set inline_styles to false so that markup formatting (underline, strikethrough) gets formatted using HTML tags (<strike>, etc.) instead of CSS styles ("text-decoration: line-through", etc.)
inputs/VegBIEN/TWiki/data/TWiki/TinyMCEPlugin.txt: theme_advanced_buttons1: added underline, strikethrough buttons
put inputs/VegBIEN/TWiki/data/TWiki/TinyMCEPlugin.txt under version control
/run: export_ (): pass "$@" to invoked functions since it may contain psql connection or config info
lib/util.sh: pg_* (): pass "$@" to invoked functions since it may contain psql connection or config info
bugfix: lib/util.sh: pg_header (): use limit=0 to prevent COPY TO from returning all rows in the table (not an issue for empty tables!)
lib/util.sh: pg_copy_to (): support sources besides whole tables (such as custom queries and tables with columns). support limiting the # of rows retrieved when using whole tables.
lib/runscripts/import.run: added export_ () command, which is run after import in all ()
bin/psql_verbose_vegbien: also echo queries issued by backslash commands (--echo-hidden)
/run: make config vars such as $schema overridable by the environment, to allow pointing the export script at a different public schema version
inputs/input.Makefile: SVN: add, %/add: */logs: also svn:ignore *.gz, used for compressed log files
added inputs/HIBG/Specimen/header.csv
added inputs/HVAA/Specimen/Herbario_occur_1360871068.csv.md5
exports/: svn:ignore *.csv
added /run, whose export_ command exports DB tables (currently the geoscrub_input table) to CSV. column information is included in the filename when the export format is specified not to contain a header row.
lib/runscripts/local.run: moved log_sql (), pg_* () to lib/util.sh because these do not depend on local external scripts in bin/
lib/runscripts/local.run: added pg_copy_to (), pg_header (), pg_export_table_no_header (), pg_export_table_to_dir_no_header ()
lib/runscripts/local.run: added mk_schema_esc, mk_table_esc aliases as shortcuts for `local *_esc; mk_esc_name *`
lib/util.sh: self_not_included (): added support for loading newly-defined aliases for use in functions in the same file, and documentation for how to do this
lib/runscripts/local.run: psql (): output only the query results to stdout (using --output), and redirect everything else to stderr
added /exports/
lib/runscripts/local.run: added log_sql (). mysql (): don't echo SQL commands when not in verbose mode. this is needed to support commands that use the query result in a $() expression, and should not have echoed commands cluttering up stdout. (unfortunately, the SQL commands are echoed to stdout rather than stderr.)
schemas/vegbien.sql: range_modeling_input: use analytical_stem instead of analytical_stem_view because it takes a long time (~2 min) to get a row from the left-joined view, due to the large number of joins and large number of rows in the joined tables
web/links/index.htm: updated to Firefox bookmarks. grouped related items together.
web/links/index.htm: author comments at beginning of bookmark desc: start with a line containing only dashes (-) rather than a blank line, because a newline will be trimmed away on reimport into Firefox, such that reexporting it will lose the author comment indicator
lib/Firefox_bookmarks.reformat.csv: label page's description: support starting a description with a line containing only dashes (-) to indicate author comments. this is better than a newline, because a newline will be trimmed away on reimport into Firefox, such that reexporting it will lose the author comment indicator
lib/Firefox_bookmarks.reformat.csv: add hyperlinked HTML anchors for folders: switch back to using <a href=...> instead of onclick, but put the <a> tag around the <H3> rather than inside it so that Firefox can correctly parse out the folder name when importing the bookmarks
lib/Firefox_bookmarks.reformat.csv: add hyperlinked HTML anchors for folders: use `onclick="javascript:document.location = '#...'"` instead of <a href=...> to allow clicking the header to go to the anchor, so that the <a> tag doesn't prevent Firefox from parsing the folder name
bugfix: lib/Firefox_bookmarks.reformat.csv: add hyperlinked HTML anchors for folders: moved <a name=.../> outside the <H3> so that Firefox can correctly parse out the folder name when importing the bookmarks
lib/Firefox_bookmarks.reformat.csv: indexing: add hyperlinked HTML anchors for folders
schemas/vegbien.sql: added range_modeling_input view
schemas/vegbien.sql: analytical_stem_view: order by datasource name so that the rows have a predictable rather than random ordering
bugfix: lib/util.sh: moved mysql (), psql () to local.run because they use local commands that are only available after local.run sets the PATH. retain a mysql () function in util.sh that adds the --verbose option to mysql.
lib/util.sh: removed "$bin_dir"/ before commands because the bin dir is now in the PATH
lib/runscripts/table.run: moved general DB commands to lib/util.sh
moved lib/*.run into runscripts/ subdir so runscripts are grouped together and easier to find rather than being scattered throughout lib/
bugfix: lib/util.sh: canon_rel_path (): use $(pwd -P) instead of $PWD because symlinks are resolved by realpath (readlink -f), so they also need to be resolved in the current dir, but $PWD does not contain the symlinks-resolved version of the current dir
inputs/GBIF/MySQL_export: use lib/util.sh instead of util.run now that the non-runscript-specific functions have been separated out into it. this ensures that MySQL_export is not unintentionally treated as a runscript (which functions differently from a normal shell script).
/.htaccess: use canonical URL without symlinks
added /.htaccess to create the right self-referential URL since / is outside the document root (linked from /vegbiendev/fs/)
schemas/VegCore/VegCore.ERD.mwb: TNRS taxonomic scrubbing steps: moved labels to the start of the arrow they label, so that the user can more easily follow the arrows from step to step by looking for an outward-pointing arrow with a label
added inputs/GBIF/_MySQL/MySQL.data.sql.gz.md5
bugfix: lib/common.Makefile: Compression: %.gz <-> %: only run command if target does not exist, to avoid overwriting the target when the compressed or uncompressed version is newer than it. the difference in mtimes can arise when one file is created after the other, and should not cause the opposite operation to be performed to try to make the other file up-to-date (leading to an infinite back-and-forth of creating one file from the other).
web/links/index.htm: updated to Firefox bookmarks. the VegPath link now includes the new favicon.
