Project

General

Profile

Statistics
| Revision:

# Date Author Comment
8893 05/09/2013 02:31 AM Aaron Marcuse-Kubitza

renamed lib/runscripts/local.run to lib/local.sh since the things it defines are not just for runscripts

8892 05/09/2013 02:22 AM Aaron Marcuse-Kubitza

lib/runscripts/local.run: added mysqldump_local, which uses vegbiendev

8891 05/09/2013 02:21 AM Aaron Marcuse-Kubitza

lib/runscripts/local.run: use postgres_compat mode because all our MySQL schemas should be usable as inputs to my2pg

8890 05/09/2013 02:20 AM Aaron Marcuse-Kubitza

lib/runscripts/local.run: added connection vars for local (bien) user

8889 05/09/2013 02:18 AM Aaron Marcuse-Kubitza

lib/util.sh: moved unalias inside include guard since it is not needed by anything before the include guard (even though it is used by commands outside the include guard, i.e. when the include guard is temporarily closed and reopened to load/unload aliases)

8888 05/09/2013 02:06 AM Aaron Marcuse-Kubitza

lib/util.sh: use declare instead of local in aliases so that the aliases can also be used outside a function (declare will create a local var when used inside a function, and a global var otherwise)

8887 05/09/2013 02:00 AM Aaron Marcuse-Kubitza

lib/util.sh: zip: ignore "zip has nothing to do" exit status as this is not an error

8886 05/09/2013 01:58 AM Aaron Marcuse-Kubitza

lib/util.sh: added exception-handling (i.e. error-suppressing) functions

8885 05/09/2013 01:03 AM Aaron Marcuse-Kubitza

bugfix: lib/util.sh: echo_func alias: moved it outside the include guard because the unalias statement is needs to be outside the include guard, and therefore so does the alias statement in order to always restore the alias

8884 05/09/2013 12:59 AM Aaron Marcuse-Kubitza

lib/util.sh: renamed echo_func_ () back to echo_func () and instead add an unalias statement before the if statement containing the function definition. this is necessary because if statements are "compound commands", which have aliases expanded when their definition is read (i.e. at the beginning of the if statement, for the entire if statement), not when they are executed, and any aliases for function names in them need to be unaliased before the if statement containing the function definition. this is necessary even if the if statement is not executed, because the expansion of such aliases will cause syntax errors in the parsing of the if statement itself....

8883 05/09/2013 12:45 AM Aaron Marcuse-Kubitza

lib/util.sh: added unalias () override, which doesn't produce an error if the alias is undefined

8882 05/09/2013 12:04 AM Aaron Marcuse-Kubitza

bugfix: lib/util.sh: renamed echo_func () to echo_func_ (), because you apparently can't have a function with the same name as an alias, even if the function is defined before the alias and an include guard prevents the function from being defined again after the alias is set. there may be some kind of alias-expanding preprocessing pass that bash only does in script mode, or maybe there is some other problem related to our use of aliases inside if statements.

8881 05/08/2013 08:02 PM Aaron Marcuse-Kubitza

*{.sh,run}: changed echo_func to an alias that includes the "$", so that callers don't need to specify the "$" manually. although the original echo_func function is still there, callers need to switch over to the alias at the same time as it's defined because otherwise the "$@" would be doubled, since echo_func refers to the alias instead.

8880 05/08/2013 07:21 PM Aaron Marcuse-Kubitza

lib/util.sh: added set_make_vars alias, for use with make-target-style shell functions

8879 05/08/2013 06:37 PM Aaron Marcuse-Kubitza

lib/util.sh: mysql_cmd (): interpret an empty string for $user, $database as if they were unset

8878 05/08/2013 06:34 PM Aaron Marcuse-Kubitza

bugfix: lib/util.sh: mysql_cmd (): add --tables after specifying --databases so that future unnamed arguments are correctly interpreted as tables

8877 05/08/2013 06:33 PM Aaron Marcuse-Kubitza

bugfix: lib/util.sh: mysql_cmd (): need to use `--databases ...` rather than `--database=...` to specify the database as a named argument

8876 05/08/2013 06:26 PM Aaron Marcuse-Kubitza

bugfix: lib/util.sh: mysql_cmd (): need space before $database in command

8875 05/08/2013 06:23 PM Aaron Marcuse-Kubitza

lib/util.sh: mysql_cmd (): use $schema as $database when provided

8874 05/08/2013 06:14 PM Aaron Marcuse-Kubitza

lib/util.sh: mysql_cmd (): accept $database connection var

8873 05/08/2013 06:02 PM Aaron Marcuse-Kubitza

*{.sh,run}: use just env instead of echo_run env now that env is an auto-echoing alias

8872 05/08/2013 05:56 PM Aaron Marcuse-Kubitza

lib/util.sh: automatically echo commands that use env

8871 05/08/2013 05:55 PM Aaron Marcuse-Kubitza

lib/util.sh: moved strings after verbose output so strings commands can use verbose output aliases

8870 05/08/2013 05:28 PM Aaron Marcuse-Kubitza

lib/util.sh: mysql_cmd (): don't ssh if the ssh_server is this machine

8869 05/08/2013 05:27 PM Aaron Marcuse-Kubitza

lib/util.sh: mysql_cmd (): allow specifying the ssh_dest as a separate ssh_server and ssh_user

8868 05/08/2013 05:08 PM Aaron Marcuse-Kubitza

lib/util.sh: using prefixed connection vars: added use_local, use_local_remote aliases

8867 05/08/2013 04:42 PM Aaron Marcuse-Kubitza

lib/util.sh: added mysqldump_diffable ()

8866 05/08/2013 04:41 PM Aaron Marcuse-Kubitza

lib/util.sh: added mysqldump () wrapper, which does what my2pg_export does, but integrates with mysql_cmd and allows turning off postgres-compatible mode

8865 05/08/2013 04:15 PM Aaron Marcuse-Kubitza

lib/util.sh: mysql_cmd (): use ${FUNCNAME1} to get the caller's command name instead of requiring the caller to pass the command name explicitly

8864 05/08/2013 03:56 PM Aaron Marcuse-Kubitza

lib/util.sh: databases: added use_remote alias to use the remote_* connection vars (which uses new import_vars)

8863 05/08/2013 03:54 PM Aaron Marcuse-Kubitza

lib/util.sh: added import_vars alias

8862 05/08/2013 03:53 PM Aaron Marcuse-Kubitza

lib/util.sh: mysql_cmd (): added doc comment

8861 05/08/2013 03:52 PM Aaron Marcuse-Kubitza

lib/util.sh: mysql (): use new mysql_cmd to auto-add connection/login opts when specified

8860 05/08/2013 03:50 PM Aaron Marcuse-Kubitza

lib/util.sh: added mysql_cmd ()

8859 05/08/2013 03:48 PM Aaron Marcuse-Kubitza

lib/util.sh: added local_inv alias

8858 05/08/2013 03:34 PM Aaron Marcuse-Kubitza

lib/util.sh: zip/unzip_newer: support extracting/updating the file even if it has an older mtime by setting $force, using new set_inv to get a var to use to determine whether to include the -u flag

8857 05/08/2013 03:32 PM Aaron Marcuse-Kubitza

lib/util.sh: added set_inv ()

8856 05/08/2013 03:30 PM Aaron Marcuse-Kubitza

lib/util.sh: echo_vars (): prefix the declare output with "+ " because it's actually a command, not just an indication of the var's value

8855 05/08/2013 03:28 PM Aaron Marcuse-Kubitza

lib/util.sh: added sed ()

8854 05/08/2013 03:23 PM Aaron Marcuse-Kubitza

lib/util.sh: added section labels

8853 05/08/2013 10:57 AM Aaron Marcuse-Kubitza

web/links/index.htm: removed broken favicons

8852 05/08/2013 10:55 AM Aaron Marcuse-Kubitza

web/links/index.htm: removed broken favicons. this requires using the SQLite Manager Firefox add-on <https://addons.mozilla.org/en-US/firefox/addon/sqlite-manager/?src=search> to manually remove the favicon entries.

8851 05/08/2013 10:18 AM Aaron Marcuse-Kubitza

lib/Firefox_bookmarks.reformat.csv: formatting fixes: don't put the src attribute of the favicon image on its own line, because it no longer contains long inline image data

8850 05/08/2013 10:15 AM Aaron Marcuse-Kubitza

lib/Firefox_bookmarks.reformat.csv: favicon image: remove ICON attributes with inline image data, since these are no longer needed for the favicon images and this avoids cluttering up the svn diff

8849 05/08/2013 10:11 AM Aaron Marcuse-Kubitza

lib/Firefox_bookmarks.reformat.csv: favicon image: use ICON_URI instead of ICON to fetch the favicon directly from the original site. this allows the favicons to be cached separately from the bookmarks page, so that the inline favicon contents don't need to be re-retrieved every time the bookmarks page is updated. this also avoids cluttering up the svn diff.

8848 05/08/2013 10:01 AM Aaron Marcuse-Kubitza

web/links/index.htm: updated to Firefox bookmarks. added description of the formats supported by PostgreSQL's COPY command and their escape sequences.

8847 05/08/2013 08:44 AM Aaron Marcuse-Kubitza

bugfix: schemas/vegbien.sql: geoscrub_input_view: only include places whose country is not NULL, because at least the country is required for geoscrubbing

8846 05/08/2013 08:34 AM Aaron Marcuse-Kubitza

/README.TXT: Schema changes: Reinstall staging tables: added step to run `make schemas/public/install` after the datasources are installed. note that this must be run after the datasources are installed, because views in public depend on some of the datasources.

8845 05/06/2013 07:10 AM Aaron Marcuse-Kubitza

/README.TXT: Schema changes: Reinstall staging tables: use `make db` instead of mk_db, which includes the required schemas/install command. some datasources use the util schema in their postprocess scripts, so this must be installed before the staging tables.

8844 05/06/2013 07:05 AM Aaron Marcuse-Kubitza

bugfix: /Makefile: moved schemas/install from install to db so that it is also run whenever the DB is reinstalled. schemas/install is needed by the staging tables reinstallation performed by reinstall_all.

8843 05/06/2013 06:33 AM Aaron Marcuse-Kubitza

bugfix: /README.TXT: Schema changes: Reinstall staging tables: also need to run `make mk_db` before running reinstall_all

8842 05/06/2013 06:30 AM Aaron Marcuse-Kubitza

/README.TXT: Schema changes: Reinstall staging tables: drop the saved previous DB at the end of the operation

8841 05/06/2013 06:18 AM Aaron Marcuse-Kubitza

/README.TXT: Schema changes: Reinstall staging tables: run all of the local machine steps on vegbiendev, including the vegbien DB rename, to ensure that the existing data is saved in case the reinstall has errors

8840 05/06/2013 05:57 AM Aaron Marcuse-Kubitza

/README.TXT: Schema changes: Reinstall staging tables: added step to first rename existing vegbien database, so it is not modified in the reinstallation

8839 05/06/2013 05:27 AM Aaron Marcuse-Kubitza

inputs/FIA/: archived no longer used BIEN2 FIA data from the nimoy geoscrub DB

8838 05/06/2013 05:26 AM Aaron Marcuse-Kubitza

inputs/FIA/_archive/: moved FIA_COND_unique, Organism into 2011-10-17/ subdir

8837 05/06/2013 04:44 AM Aaron Marcuse-Kubitza

/README.TXT: Schema changes: Reinstall staging tables: added step to first run this on the local machine to check for any errors, before running it on the live DB, to prevent the live DB from becoming unrestorable in the case of bugs

8836 05/06/2013 04:27 AM Aaron Marcuse-Kubitza

bugfix: bin/with_all: pipe yes to the command in case it makes any interactive prompts, as in e.g. the TNRS staging tables reinstall

8835 05/06/2013 04:25 AM Aaron Marcuse-Kubitza

bugfix: inputs/input.Makefile: sql/install: manually specify $no_search_path option to psql_script_vegbien, which is added automatically in $(psqlNoSearchPath) but that uses psql_verbose_vegbien

8834 05/06/2013 04:12 AM Aaron Marcuse-Kubitza

bugfix: *{.sh,run}: use `local` when setting default values for params to prevent these values from leaking into the calling context, where it may have unexpected effects (e.g. pg_copy_to would permanently set $source, which would then be used on the next invocation of pg_copy_to instead of regenerating $source from the next $table value). this involves using ${var-default} or ${var:-default} to return the value to the local statement instead of setting it directly with ${var=default} or ${var:=default}.

8833 05/06/2013 04:00 AM Aaron Marcuse-Kubitza

*{.sh,run}: commands that take named arguments: usage messages: changed to use simpler bash syntax for specifying command-specific env vars, which does not require a subshell ("The environment for any simple command or function may be augmented temporarily by prefixing it with parameter assignments" <http://www.gnu.org/software/bash/manual/bash.html#Environment&gt;)

8832 05/06/2013 03:14 AM Aaron Marcuse-Kubitza

schemas/pg_hba*.conf: removed trailing whitespace

8831 05/06/2013 03:10 AM Aaron Marcuse-Kubitza

schemas/pg_hba*.conf: allow the bien group to access all databases, including test (on the Mac). note that this does not grant access to the vegbank DB on vegbiendev, because that is owned by postgres.

8830 05/06/2013 02:32 AM Aaron Marcuse-Kubitza

web/links/index.htm: explicitly separate page's description from author comments where needed

8829 05/06/2013 02:29 AM Aaron Marcuse-Kubitza

web/links/index.htm: use - instead of --- to separate the page's description from the author comments because it's faster to type and one hyphen is sufficient

8828 05/06/2013 02:21 AM Aaron Marcuse-Kubitza

lib/Firefox_bookmarks.reformat.csv: labeling page's description: don't match descriptions with a leading blank line as an author comment, because sometimes page descriptions themselves start with a blank line. instead, author comments at the beginning of bookmark descriptions should always start with a " or be preceded by a line containing only hyphens (-). note that it was never reliable to use a blank line for this purpose, because it would be trimmed out upon reimport of the HTML file into Firefox.

8827 05/06/2013 02:17 AM Aaron Marcuse-Kubitza

web/links/index.htm: updated to Firefox bookmarks

8826 05/05/2013 10:35 PM Aaron Marcuse-Kubitza

inputs/NVS/*/map.csv: Taxon Growth Form: use new _lowercase() to avoid needing to manually map each value that is already valid but just needs to be lowercased

8825 05/05/2013 10:28 PM Aaron Marcuse-Kubitza

schemas/util.sql: added _lowercase()

8824 05/05/2013 09:05 PM Aaron Marcuse-Kubitza

added inputs/CTFS/ERD/ VegPath location

8823 05/05/2013 11:57 AM Aaron Marcuse-Kubitza

added inputs/MO/_src/_README.TXT with warning that the refresh provided to us is missing several columns from the original extract: CollectionCode, CatalogNumber, IndividualCount

8822 05/05/2013 10:59 AM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: reduced MaxColumnsDisplayed to 10 to prevent wide tables from taking up too much space in the ERD. this was a big problem in the VegBIEN ERD, which we are hoping to prevent in the VegCore ERD as first-class columns get added to the tables. note that it's very important to ensure that the first-class columns are sorted in importance order, so that the 10 most important are visible in the ERD.

8821 05/05/2013 10:34 AM Aaron Marcuse-Kubitza

schemas/VegCore/VegCore.ERD.mwb: renamed specimen_holder_institution to specimenholder_institution because specimenholder is one word

8820 05/05/2013 10:12 AM Aaron Marcuse-Kubitza

lib/sql_io.py: put_table(): Calling wrapper function: adding pkey or index on the resulting table: don't display warning if a pkey can't be added, because this is actually a legitimate situation when the called function is set-returning and can return multiple rows for one input. having this as a warning results in spurious warnings in the automated tests (which look confusingly like ignored errors because Python warnings include debugging context information). e.g. `make inputs/Madidi/IndividualObservation/test.by_col.xml` causes this error in the sourcelist->sourcename splitting step (which of course can produce multiple specimenholder institutions)....

8819 05/05/2013 10:00 AM Aaron Marcuse-Kubitza

bugfix: lib/sql.py: parse_exception(): typed_name_re (used by MissingCastException, etc.): don't require "" around the value, because function names in "function does not exist" exceptions are (now?) output without them. this exception did not occur in the last import, but does occur in the automated testing and when running the associated query in pgAdmin (in both Linux and Mac OS X).

8818 05/05/2013 04:30 AM Aaron Marcuse-Kubitza

bugfix: /Makefile: apache-Linux: use the standard apt-get syntax for Makefiles (`-sudo apt-get --yes install`) instead of just `apt-get install`

8817 05/05/2013 04:24 AM Aaron Marcuse-Kubitza

/Makefile: added Apache 2.4 installation

8816 05/05/2013 03:57 AM Aaron Marcuse-Kubitza

lib/util.sh: added join ()

8815 05/05/2013 02:58 AM Aaron Marcuse-Kubitza

web/.twiki/data/Main/TWikiPreferences.txt: updated with current settings

8814 05/05/2013 02:55 AM Aaron Marcuse-Kubitza

put web/.twiki/data/Main/TWikiPreferences.txt under version control. note that this file (Main/TWikiPreferences.txt) is the correct file to change the settings in, not TWiki/TWikiPreferences.txt.

8813 05/05/2013 02:33 AM Aaron Marcuse-Kubitza

put web/.twiki/data/TWiki/TWikiPreferences.txt under version control

8812 05/05/2013 01:58 AM Aaron Marcuse-Kubitza

inputs/VegBIEN/TWiki/data/TWiki/TinyMCEPlugin.txt: twikibuttons_formats: added blockquote format

8811 05/05/2013 01:47 AM Aaron Marcuse-Kubitza

inputs/VegBIEN/TWiki/data/TWiki/TinyMCEPlugin.txt: set inline_styles to false so that markup formatting (underline, strikethrough) gets formatted using HTML tags (<strike>, etc.) instead of CSS styles ("text-decoration: line-through", etc.)

8810 05/05/2013 01:26 AM Aaron Marcuse-Kubitza

inputs/VegBIEN/TWiki/data/TWiki/TinyMCEPlugin.txt: theme_advanced_buttons1: added underline, strikethrough buttons

8809 05/05/2013 01:25 AM Aaron Marcuse-Kubitza

put inputs/VegBIEN/TWiki/data/TWiki/TinyMCEPlugin.txt under version control

8808 05/02/2013 11:07 PM Aaron Marcuse-Kubitza

/run: export_ (): pass "$@" to invoked functions since it may contain psql connection or config info

8807 05/02/2013 11:06 PM Aaron Marcuse-Kubitza

lib/util.sh: pg_* (): pass "$@" to invoked functions since it may contain psql connection or config info

8806 05/02/2013 10:59 PM Aaron Marcuse-Kubitza

bugfix: lib/util.sh: pg_header (): use limit=0 to prevent COPY TO from returning all rows in the table (not an issue for empty tables!)

8805 05/02/2013 10:53 PM Aaron Marcuse-Kubitza

lib/util.sh: pg_copy_to (): support sources besides whole tables (such as custom queries and tables with columns). support limiting the # of rows retrieved when using whole tables.

8804 05/02/2013 10:01 PM Aaron Marcuse-Kubitza

lib/runscripts/import.run: added export_ () command, which is run after import in all ()

8803 05/02/2013 09:57 PM Aaron Marcuse-Kubitza

bin/psql_verbose_vegbien: also echo queries issued by backslash commands (--echo-hidden)

8802 05/02/2013 09:34 PM Aaron Marcuse-Kubitza

/run: make config vars such as $schema overridable by the environment, to allow pointing the export script at a different public schema version

8801 05/02/2013 08:53 PM Aaron Marcuse-Kubitza

inputs/input.Makefile: SVN: add, %/add: */logs: also svn:ignore *.gz, used for compressed log files

8800 05/02/2013 08:50 PM Aaron Marcuse-Kubitza

added inputs/HIBG/Specimen/header.csv

8799 05/02/2013 08:50 PM Aaron Marcuse-Kubitza

added inputs/HVAA/Specimen/Herbario_occur_1360871068.csv.md5

8798 05/02/2013 08:40 PM Aaron Marcuse-Kubitza

exports/: svn:ignore *.csv

8797 05/02/2013 08:39 PM Aaron Marcuse-Kubitza

added /run, whose export_ command exports DB tables (currently the geoscrub_input table) to CSV. column information is included in the filename when the export format is specified not to contain a header row.

8796 05/02/2013 08:35 PM Aaron Marcuse-Kubitza

lib/runscripts/local.run: moved log_sql (), pg_* () to lib/util.sh because these do not depend on local external scripts in bin/

8795 05/02/2013 08:27 PM Aaron Marcuse-Kubitza

lib/runscripts/local.run: added pg_copy_to (), pg_header (), pg_export_table_no_header (), pg_export_table_to_dir_no_header ()

8794 05/02/2013 08:26 PM Aaron Marcuse-Kubitza

lib/runscripts/local.run: added mk_schema_esc, mk_table_esc aliases as shortcuts for `local *_esc; mk_esc_name *`