lib/sh/util.sh: get_stack_frame_after(): renamed to skip_stack_frames() for clarity

lib/sh/util.sh: added skip_stack_frame_in_caller, unskip_stack_frame_in_caller

lib/sh/util.sh: added prev_stack_frame

lib/sh/util.sh: echo_func(): use new format_stack_frame, which adds call context information to what was provided by func_loc

fix: lib/sh/util.sh: format_stack_frame(): need to hide canon_rel_path() info using log+

lib/sh/util.sh: added get_stack_frame_after()

lib/sh/util.sh: added matches()

lib/sh/util.sh: next_stack_frame: documented usage

fix: lib/runscripts/util.run: runscript template: all(): moved example commands to target(), where they would more likely be located

lib/sh/util.sh: format_stack_frame(): support including args

lib/sh/util.sh: debugging: added pst() (print_stack_trace)

lib/sh/util.sh: added stack_trace(), print_stack_trace()

lib/sh/util.sh: added format_stack_frame()

lib/sh/util.sh: added get_stack_frame() and helpers

lib/sh/util.sh: terminal: moved before errors so it can be used by it

lib/sh/util.sh: errors, debugging: moved after datatype sections so their functions can use these

bugfix: lib/sh/util.sh: canon_rel_path() stub: proper no-op requires passing through original path

lib/sh/util.sh: canon_rel_path(): fall back to original path if can't resolve, instead of errexiting

bugfix: lib/sh/util.sh: canon_rel_path(): don't re-localize $path because this clears it

lib/sh/util.sh: canon_rel_path(): import $1 to $path before function body, so that the function body can be moved to a nested function

lib/sh/util.sh: added canon_rel_path() stub for use by debugging functions

lib/sh/util.sh: moved func_loc() to before debugging section so it can be used by debugging functions

bugfix: lib/sh/util.sh: command__exec(): need to restore $verbosity before calling die_e

fix: lib/sh/local.sh: $sync_remote_url: need $USER so user can be overridden when running as root

lib/Firefox_bookmarks.reformat.csv: label page's self-description as such: also support quotations enclosed in '

lib/sh/util.sh: echo_vars(): merge repeated flags so there aren't flags in between the vars (which is also not valid declare syntax)

lib/sh/db.sh: pg_cmd(): log vars on same line to avoid clutter

lib/sh/util.sh: echo_vars(): put all the vars on the same line so they don't clutter up the call graph generated at the default verbosity

lib/tnrs.py single_tnrs_request(), bin/tnrs_client: use_tnrs_export: default to False because this mode uses incorrect selected matches (vegpath.org/issues/943), and the JSON mode that fixes this is now available

bugfix: lib/csvs.py: JsonReader: need to pass col_order to row_dict_to_list_reader

bugfix: lib/tnrs.py: JSON output: need to stringify arrays so they match what is output in TSV-export mode

lib/csvs.py: JsonReader: added support for values that are arrays

lib/csvs.py: MultiFilter: inherit from WrapReader instead of Filter to avoid needing to define a no-op filter_() function

bugfix: lib/csvs.py: row_dict_to_list_reader: need to override next() directly instead of just using Filter, because Filter doesn't support returning multiple rows for one input row (in this case, prepending a header row). this caused the 1st data row to be missing.

lib/csvs.py: Filter: inherit from WrapReader, which separates out the CSV-reader API code

lib/csvs.py: added WrapReader

lib/csvs.py: added Reader

lib/csvs.py: JsonReader: factored out row-dict-to-list into new row_dict_to_list_reader so that JSON-specific preprocessing is kept separate from the row format translation

lib/csvs.py: added MultiFilter, which enables applying multiple filters by nesting

lib/tnrs.py: single_tnrs_request(): JSON mode: implemented output of JSON data

lib/tnrs.py: single_tnrs_request(): factored out wrapping in TnrsOutputStream, since this is done for both modes

fix: lib/tnrs.py: JSON mode: TSV export columns: need to translate these to JSON column names before they can be used with the JSON data

lib/csvs.py: added JsonReader, which reads parsed JSON data as row tuples

lib/csvs.py: added row_dict_to_list(), which translates a CSV dict-based row to a list-based one

lib/csvs.py: RowNumFilter: added support for filtering the header row as well

lib/csvs.py: ColInsertFilter: added support for filtering the header row as well

lib/csvs.py: InputRewriter: documented that this is also a stream (in addition to inheriting from StreamFilter)

bugfix: lib/csvs.py: InputRewriter: accept a reader, as would be expected, instead of a custom stream whose lines are tuples

fix: lib/sql_io.py: append_csv(): use new csvs.ProgressInputFilter instead of streams.ProgressInputStream(csvs.StreamFilter(__)), so that the input to csvs.InputRewriter is a reader, not a stream. this avoids the need for csvs.InputRewriter to accept a stream whose lines are tuples, instead of the expected reader.

lib/csvs.py: added ProgressInputFilter, analogous to streams.ProgressInputStream

lib/sql_io.py: added commented-out debug statement used to troubleshoot copy_expert() errors

lib/dicts.py: added pair_keys(), pair_values()

bugfix: lib/streams.py: CaptureStream: end_idx must also be > start_idx

lib/tnrs.py: single_tnrs_request(): use_tnrs_export=False: need to obtain export columns

lib/csvs.py: added header(stream)

fix: lib/tnrs.py: single_tnrs_request(): need to `assert name_ct >= 1`, because with no names, TNRS hangs indefinitely

bugfix: lib/sh/archives.sh: compress(): don't include dir prefix in zip archive

lib/sh/util.sh: cd(): use echo_run instead of a manual echo_cmd call

fix: lib/sh/util.sh: cd(): indent after running cd rather than before

lib/sh/util.sh: cd(): support rebasing path vars for the new dir

bugfix: lib/sh/archives.sh: compress(): need to use zip's path syntax to avoid the file in the archive being named "-"

lib/tnrs.py: added option to avoid using TNRS's TSV export feature, which currently returns incorrect selected matches (vegpath.org/issues/943). this has been implemented up through the GWT/JSON decoding.

lib/tnrs.py: added gwt_decode()

lib/strings.py: added unesc_quotes() and helper functions

lib/strings.py: added json_decode()

lib/runscripts/extract.run: export_(): also compress created file

lib/sh/archives.sh: added compress(), expand(), which handle compression of individual files

lib/tnrs.py: documentation about output of the retrieve step: added that this is also unusable because the array does not contain all the columns and contains no column names

fix: lib/tnrs.py: retrieval_request_template: source_sorting (Constrain by Source): corrected explanation to reflect that the behavior is actually the same in both modes, since only one match is ever marked as selected, and that match should always come first

bugfix: lib/sh/util.sh: str2varname(): need to lowercase str because on case-insensitive filesystems, paths sometimes canonicalize to a different capitalization than the original

lib/sh/util.sh: added lowercase()

bugfix: lib/sh/util.sh: die(): need stub since this is invoked before it's defined

bugfix: lib/sh/util.sh: setup_log_fd(): don't change $log_fd to stdlog until stdlog is set up, to avoid "Bad file descriptor" errors

lib/sh/util.sh: func_override(), copy_func(): added echo_func to facilitate debugging

bugfix: lib/sh/util.sh: stubs: log++ alias also needs to be moved to stub section

bugfix: lib/Firefox_bookmarks.reformat.csv: URLs: match only the uppercase tags used by Firefox, not any lowercase tags added by the user

fix: lib/Firefox_bookmarks.reformat.csv: page's self-description: updated comment to match regexp

bugfix: lib/Firefox_bookmarks.reformat.csv: page's self-description: updated "page's self-description: " prefix to remove

lib/sh/sync.sh: db_snapshot(): before backing up, trim bloated temp files (eg. from rolled back changes)

bugfix: lib/sql_io.py: put_table(): handle_MissingCastException(): when updating join_cols, don't add new entry for join_cols[out_col], only update existing one. this fixes #902 (import bug), and with #902 fixed, #887 (disk space leak) should no longer occur.

bugfix: lib/Firefox_bookmarks.reformat.csv: updated for new Firefox bookmarks format, which indents the <DD> tag

lib/tnrs.py: dirty: documented that this actually used to be on in the web app (see r9910, 2013-6-18), but does not appear to be needed (the source_sorting bug alluded to in r9910 is not fixed by enabling the dirty setting)

lib/tnrs.py: requests: also debug-print request URL

lib/tnrs.py: Download: include the same debug info as do_request()

lib/tnrs.py: do_request(): also debug-print request headers

lib/tnrs.py: download_request_template: dirty: documented why this must be off

bugfix: lib/tnrs.py: download_request_template: fixed bug where multiple names were being marked as Selected, because dirty was incorrectly set to true unlike in the web app

fix: lib/phpPgAdmin.login.php.diff: use relative file path rather than the path the file was at when the patch was created

/Makefile, lib/phpPgAdmin.login.php.diff: public_ user: added auto-filled password so that users would not be confused as to what to type in the password field

lib/tnrs.py: source_sorting (Constrain by Source): documented the different behavior for this in each match mode (all-matches and best-match)

fix: lib/runscripts/extract.run: export_(): explicitly prevent files from becoming web-accessible, to protect against an incorrect umask in the calling process

lib/tnrs.py: max_names: raised back up to 500 now that a workaround for the Internal Server Errors is in place (https://github.com/iPlantCollaborativeOpenSource/TNRS/issues/7)

fix: lib/tnrs.py: max_names: lowered to 50 because the dev TNRS server is now always crashing with an Internal Server Error when scrubbing 500 names at a time (https://github.com/iPlantCollaborativeOpenSource/TNRS/issues/7)

fix: lib/tnrs.py: Constrain by Source: turn it on so that the download settings reflect what TNRS actually used, while this is broken

fix: lib/tnrs.py: max_names: reduced back to 500 because even 5000 crashes the dev TNRS server

lib/tnrs.py: max_names: reduced to 5000 because 100,000 causes an internal server error

lib/tnrs.py: switched to downloading all matches per name, as is needed to implement #917. note that this will break the parts of the schema that use the tnrs table, until Brad's match-picking algorithm can be implemented, but this tradeoff is necessary to be able to begin scrubbing sooner (Martha; wiki.vegpath.org/2014-05-29_conference_call#TNRS)

lib/tnrs.py: max_names: increased to 100000 because the dev server can handle more names (no simultaneous users), as decided in the conference call (wiki.vegpath.org/2014-05-29_conference_call#TNRS)

fix: lib/PostgreSQL-MySQL.csv: need to replace "double precision" with "double" to work with MySQL Workbench 5.2.47

lib/tnrs.py: commented out the value of max_names that is not active, for clarity