lib/sql_io.py: put_table(): Calling wrapper function: adding pkey or index on the resulting table: don't display warning if a pkey can't be added, because this is actually a legitimate situation when the called function is set-returning and can return multiple rows for one input. having this as a warning results in spurious warnings in the automated tests (which look confusingly like ignored errors because Python warnings include debugging context information). e.g. `make inputs/Madidi/IndividualObservation/test.by_col.xml` causes this error in the sourcelist->sourcename splitting step (which of course can produce multiple specimenholder institutions)....
bugfix: lib/sql.py: parse_exception(): typed_name_re (used by MissingCastException, etc.): don't require "" around the value, because function names in "function does not exist" exceptions are (now?) output without them. this exception did not occur in the last import, but does occur in the automated testing and when running the associated query in pgAdmin (in both Linux and Mac OS X).
bugfix: /Makefile: apache-Linux: use the standard apt-get syntax for Makefiles (`-sudo apt-get --yes install`) instead of just `apt-get install`
/Makefile: added Apache 2.4 installation
lib/util.sh: added join ()
web/.twiki/data/Main/TWikiPreferences.txt: updated with current settings
put web/.twiki/data/Main/TWikiPreferences.txt under version control. note that this file (Main/TWikiPreferences.txt) is the correct file to change the settings in, not TWiki/TWikiPreferences.txt.
put web/.twiki/data/TWiki/TWikiPreferences.txt under version control
inputs/VegBIEN/TWiki/data/TWiki/TinyMCEPlugin.txt: twikibuttons_formats: added blockquote format
inputs/VegBIEN/TWiki/data/TWiki/TinyMCEPlugin.txt: set inline_styles to false so that markup formatting (underline, strikethrough) gets formatted using HTML tags (<strike>, etc.) instead of CSS styles ("text-decoration: line-through", etc.)
inputs/VegBIEN/TWiki/data/TWiki/TinyMCEPlugin.txt: theme_advanced_buttons1: added underline, strikethrough buttons
put inputs/VegBIEN/TWiki/data/TWiki/TinyMCEPlugin.txt under version control
/run: export_ (): pass "$@" to invoked functions since it may contain psql connection or config info
lib/util.sh: pg_* (): pass "$@" to invoked functions since it may contain psql connection or config info
bugfix: lib/util.sh: pg_header (): use limit=0 to prevent COPY TO from returning all rows in the table (not an issue for empty tables!)
lib/util.sh: pg_copy_to (): support sources besides whole tables (such as custom queries and tables with columns). support limiting the # of rows retrieved when using whole tables.
lib/runscripts/import.run: added export_ () command, which is run after import in all ()
bin/psql_verbose_vegbien: also echo queries issued by backslash commands (--echo-hidden)
/run: make config vars such as $schema overridable by the environment, to allow pointing the export script at a different public schema version
inputs/input.Makefile: SVN: add, %/add: */logs: also svn:ignore *.gz, used for compressed log files
added inputs/HIBG/Specimen/header.csv
added inputs/HVAA/Specimen/Herbario_occur_1360871068.csv.md5
exports/: svn:ignore *.csv
added /run, whose export_ command exports DB tables (currently the geoscrub_input table) to CSV. column information is included in the filename when the export format is specified not to contain a header row.
lib/runscripts/local.run: moved log_sql (), pg_* () to lib/util.sh because these do not depend on local external scripts in bin/
lib/runscripts/local.run: added pg_copy_to (), pg_header (), pg_export_table_no_header (), pg_export_table_to_dir_no_header ()
lib/runscripts/local.run: added mk_schema_esc, mk_table_esc aliases as shortcuts for `local *_esc; mk_esc_name *`
lib/util.sh: self_not_included (): added support for loading newly-defined aliases for use in functions in the same file, and documentation for how to do this
lib/runscripts/local.run: psql (): output only the query results to stdout (using --output), and redirect everything else to stderr
added /exports/
lib/runscripts/local.run: added log_sql (). mysql (): don't echo SQL commands when not in verbose mode. this is needed to support commands that use the query result in a $() expression, and should not have echoed commands cluttering up stdout. (unfortunately, the SQL commands are echoed to stdout rather than stderr.)
schemas/vegbien.sql: range_modeling_input: use analytical_stem instead of analytical_stem_view because it takes a long time (~2 min) to get a row from the left-joined view, due to the large number of joins and large number of rows in the joined tables
web/links/index.htm: updated to Firefox bookmarks. grouped related items together.
web/links/index.htm: author comments at beginning of bookmark desc: start with a line containing only dashes (-) rather than a blank line, because a newline will be trimmed away on reimport into Firefox, such that reexporting it will lose the author comment indicator
lib/Firefox_bookmarks.reformat.csv: label page's description: support starting a description with a line containing only dashes (-) to indicate author comments. this is better than a newline, because a newline will be trimmed away on reimport into Firefox, such that reexporting it will lose the author comment indicator
lib/Firefox_bookmarks.reformat.csv: add hyperlinked HTML anchors for folders: switch back to using <a href=...> instead of onclick, but put the <a> tag around the <H3> rather than inside it so that Firefox can correctly parse out the folder name when importing the bookmarks
lib/Firefox_bookmarks.reformat.csv: add hyperlinked HTML anchors for folders: use `onclick="javascript:document.location = '#...'"` instead of <a href=...> to allow clicking the header to go to the anchor, so that the <a> tag doesn't prevent Firefox from parsing the folder name
bugfix: lib/Firefox_bookmarks.reformat.csv: add hyperlinked HTML anchors for folders: moved <a name=.../> outside the <H3> so that Firefox can correctly parse out the folder name when importing the bookmarks
lib/Firefox_bookmarks.reformat.csv: indexing: add hyperlinked HTML anchors for folders
web/links/index.htm: updated to Firefox bookmarks
schemas/vegbien.sql: added range_modeling_input view
schemas/vegbien.sql: analytical_stem_view: order by datasource name so that the rows have a predictable rather than random ordering
bugfix: lib/util.sh: moved mysql (), psql () to local.run because they use local commands that are only available after local.run sets the PATH. retain a mysql () function in util.sh that adds the --verbose option to mysql.
lib/util.sh: removed "$bin_dir"/ before commands because the bin dir is now in the PATH
lib/runscripts/table.run: moved general DB commands to lib/util.sh
moved lib/*.run into runscripts/ subdir so runscripts are grouped together and easier to find rather than being scattered throughout lib/
bugfix: lib/util.sh: canon_rel_path (): use $(pwd -P) instead of $PWD because symlinks are resolved by realpath (readlink -f), so they also need to be resolved in the current dir, but $PWD does not contain the symlinks-resolved version of the current dir
inputs/GBIF/MySQL_export: use lib/util.sh instead of util.run now that the non-runscript-specific functions have been separated out into it. this ensures that MySQL_export is not unintentionally treated as a runscript (which functions differently from a normal shell script).
/.htaccess: use canonical URL without symlinks
added /.htaccess to create the right self-referential URL since / is outside the document root (linked from /vegbiendev/fs/)
schemas/VegCore/VegCore.ERD.mwb: TNRS taxonomic scrubbing steps: moved labels to the start of the arrow they label, so that the user can more easily follow the arrows from step to step by looking for an outward-pointing arrow with a label
added inputs/GBIF/_MySQL/MySQL.data.sql.gz.md5
bugfix: lib/common.Makefile: Compression: %.gz <-> %: only run command if target does not exist, to avoid overwriting the target when the compressed or uncompressed version is newer than it. the difference in mtimes can arise when one file is created after the other, and should not cause the opposite operation to be performed to try to make the other file up-to-date (leading to an infinite back-and-forth of creating one file from the other).
web/links/index.htm: updated to Firefox bookmarks. the VegPath link now includes the new favicon.
added web/favicon.ico generated from schemas/BIEN_logo.png
web/main.conf: added SSL-protected site for use with password-protected areas of the site, so that the user's password is not transmitted in plaintext. (because the SSL certificate is self-signed, it will unfortunately display the Confirm Security Exception message in Firefox.) note that digest authentication cannot be used to avoid plaintext passwords, because it requires knowing the user's original password to generate the digest, but we have only the MD5 sums in /etc/shadow.
web/links/index.htm: put all bookmarks into one subfolder named BIEN links, to facilitate importing them all into one folder on another machine. this also helps when upgrading to a newer version of the bookmarks, because the previous version must be deleted and this now requires deleting only a single folder.
lib/Firefox_bookmarks.reformat.csv: use <H3 style="font-size: 200%;"> instead of <H1> to make the first folder's name the page title, so that Firefox can reimport the first folder as a bookmarks folder
bugfix: lib/Firefox_bookmarks.reformat.csv: make first folder name the page title: added back replacement string
web/links/index.htm.run: clean_up (): removed command to add page title because this is now done by lib/Firefox_bookmarks.reformat.csv
lib/Firefox_bookmarks.reformat.csv: use page title as window title
lib/Firefox_bookmarks.reformat.csv: remove previous page title
lib/Firefox_bookmarks.reformat.csv: make first folder name the page title
lib/Firefox_bookmarks.reformat.csv: move Bookmarks Toolbar inside first folder instead of before it. this allows storing all bookmarks in one subfolder so that when the web page is imported into another computer's bookmarks, the imported bookmarks are all together in one folder.
bugfix: lib/Firefox_bookmarks.reformat.csv: refactored regexps to account for Python applying (?s) to the entire regexp, even if it appears inside a () subgroup. the Python behavior is in contrast to the standard regexp behavior used by Java, etc. where (?s) applies only to the subgroup it appears in.
web/links/index.htm: updated to Firefox bookmarks. resources: put unsorted links into subfolders.
bugfix: lib/Firefox_bookmarks.reformat.csv: add <style>: format bookmark description as plaintext: use `white-space: pre-wrap` instead of pre in order to wrap long lines
bugfix: lib/Firefox_bookmarks.reformat.csv: favicon images: made replacements idempotent so that the favicons would not be deleted the next time the web/links/index.htm.run is run (due to a newline being added before ICON= )
lib/Firefox_bookmarks.reformat.csv: favicon images: display placeholder favicon image for every link to ensure uniform alignment
added web/links/blank.gif symlink to web/blank.gif for shorter blank.gif URLs
added web/blank.gif
lib/Firefox_bookmarks.reformat.csv: display ICON attributes as separate images. note that this uses the "data:" embedded image contents provided in the bookmarks file, so the images load instantly along with the page instead of needing to be separately retrieved from each server.
lib/Firefox_bookmarks.reformat.csv: instead of removing ICON attributes with favicon contents, put them on their own line so the rest of the line is easier to see
lib/Firefox_bookmarks.reformat.csv: divided replacements into sections
lib/Firefox_bookmarks.reformat.csv: move Bookmarks Toolbar to beginning, remove its desc, and rename it to Quick links
web/links/index.htm: removed extra divider after Recent Tags, Recently Bookmarked because section dividers interfere with regexp matching and the Bookmarks Toolbar is already visually separate from them
web/links/index.htm: moved Recent Tags, Recently Bookmarked special folders to end because they are Firefox-specific and not part of the BIEN links (they also do not work in a browser because they are Firefox-specific query bookmarks instead of actual folders)
web/links/index.htm.run: clean_up (): rename the page title from "Bookmarks" to "BIEN links"
bin/repl: skip header instead of parsing it, since its columns are not used. this also allows using just a blank line as the header when the column names are not important.
web/links/index.htm: use initial blank line instead of previous "-" workaround to indicate author comment at start of bookmark desc
web/links/index.htm: use initial blank line to indicate author comment at start of bookmark desc
lib/Firefox_bookmarks.reformat.csv: label page's description: don't label a bookmark desc starting with a blank line (indicating an author-added comment) as a page's description
lib/Firefox_bookmarks.reformat.csv: label page's description: don't label an author-added quotation (which starts with " ) as the page's description
bin/in_place: set the permissions of the new file to the original file when `chmod --reference` is available. this ensures that when a file is marked web-accessible, it stays web-accessible when you run an in_place command on it.
web/links/index.htm: BitTorrent Sync: security warning about data leak: added link to reported bug at http://forum.bittorrent.com/topic/18612-security-hole-in-linux-webui-makes-all-your-files-publicly-accessible/ . updated workaround instructions.
web/links/index.htm: updated to Firefox bookmarks. BitTorrent Sync: added security warning that running btsync on Linux will by default create a publicly-accessible, unprotected WebUI, allowing anyone on the web to create a sync folder to view and edit any of your files. to avoid this, always run btsync with a config file which sets a password on the WebUI. if you have already run btsync, run `killall btsync` to turn off the WebUI.
web/links/index.htm: updated to Firefox bookmarks. changes include links about BitTorrent Sync.
lib/Firefox_bookmarks.reformat.csv: label <meta> description: renamed label text to "page's description" to be understandable to non-programmers
lib/Firefox_bookmarks.reformat.csv: label <meta> description: remove any existing description label, to avoid duplicating the description label when the text of the description label changes
lib/Firefox_bookmarks.reformat.csv: label <meta> description as such to avoid confusing it with comments added by the bookmark author, who may not agree with it. the <meta> description is in the 1st paragraph(s), when provided.
lib/Firefox_bookmarks.reformat.csv: remove LAST_MODIFIED attributes, which seem to change whenever the favicon is updated and clutter up the svn diff
lib/Firefox_bookmarks.reformat.csv: remove ICON attributes with favicon contents, which clutter up the svn diffs
added web/links/index.htm.run with clean_up action to run whenever the index.htm bookmarks file is changed (`web/links/index.htm.run clean_up`)
added lib/Firefox_bookmarks.reformat.csv
README.TXT: Maintenance: to synchronize a Mac's settings with my testing machine's: exclude only backup files in the Documents/BIEN/svn/backups/ dir (sync the other files)
README.TXT: Maintenance: to synchronize vegbiendev, jupiter, and your local machine: when changes are made on vegbiendev: don't sync the group, which can be bien on vegbiendev but not on jupiter (where aaronmk is not a member of the bien group)
README.TXT: Maintenance: to synchronize vegbiendev, jupiter, and your local machine: when changes are made on vegbiendev: include *.md5
README.TXT: Maintenance: removed step to backup the Ubuntu VM, because this is now done as part of the "backup files not in Time Machine" step
