Activity
From 03/06/2013 to 04/04/2013
04/04/2013
- 10:01 PM Revision 8338: schemas/VegCore/VegCore.ERD.mwb: Added table colors
- 10:01 PM Revision 8337: Removed backup file schemas/VegCore/VegCore.ERD.mwb.bak
- 09:48 PM Revision 8336: Added schemas/VegCore/VegCore.ERD.mwb, VegCore.my.sql with first VegCore ERD and MySQL schema. All tables are in the ERD, but contain only pkey and fkey columns.
- 09:52 AM Revision 8335: lib/sql.py: mk_select(): using subset function: Turn off enable_sort (within the transaction) to avoid unwanted slow sorts. This change (along with the subset functions themselves) should significantly reduce the long FIA.occurrence_all table subset time (~8 hours altogether) and with it the total import time, which had more than doubled as a result of the FIA refresh. Note that this issue would have been even more pronounced for larger datasets, such as the GBIF refresh, which would have taken ~2.5 days longer (400 million rows * ~30% are plants * (FIA: ~8 hours/16.7 million rows) * 1 day/24 hours).
- 09:30 AM Revision 8334: lib/sql.py: mk_select(): Use subset function when it's available for fast querying at large OFFSET values
- 09:29 AM Revision 8333: lib/sql.py: Added has_subset_func()
- 08:48 AM Revision 8332: inputs/FIA/occurrence_all/import: Run mk_subset_by_row_num_func() to make the subset functions available for fast querying at large OFFSET values
- 08:43 AM Revision 8331: schemas/util.sql: mk_subset_by_row_num_func(): regular subset function: Fixed bug where need to add 1 to the 0-based offset_ to get the 1-based row_num (which is usually a serial column)
- 08:38 AM Revision 8330: schemas/util.sql: mk_subset_by_row_num_func(): regular subset function: Fixed bug where need to subtract 1 from the end row_num because BETWEEN limits are inclusive of the bounds
- 08:33 AM Revision 8329: schemas/util.sql: mk_subset_by_row_num_func(): regular subset function: Fixed bug where also need to COALESCE() offset_ to 0 when it's added to the limit_
- 08:20 AM Revision 8328: schemas/util.sql: mk_subset_by_row_num_func(): subset function which turns off enable_sort: Fixed bug where need to pass ($2, $3) instead of ($1, $2) to the regular subset function
- 08:14 AM Revision 8327: inputs/FIA/occurrence_all/import: Added occurrence_all-row_num column for use with mk_subset_by_row_num_func()
- 08:12 AM Revision 8326: schemas/util.sql: mk_subset_by_row_num_func(): Also create subset function which turns off enable_sort. This is used for limit values greater than ~100,000 to avoid unwanted slow sorts. The regular subset function is still needed to work with EXPLAIN, so that it produces expanded output instead of just a function scan.
- 07:27 AM Revision 8325: schemas/util.sql: Added mk_subset_by_row_num_func()
- 07:10 AM Revision 8324: schemas/util.sql: Added type_qual_name()
- 06:33 AM Revision 8323: schemas/util.sql: force_update_view(): Fixed bug where also need to drop view for "cannot change name of view column" errors
- 05:24 AM Revision 8322: inputs/FIA/occurrence_all/import: Use new force_update_view(), which only drops the view if its columns have changed and otherwise just uses CREATE OR REPLACE VIEW, rather than always first running DROP VIEW IF EXISTS
- 05:20 AM Revision 8321: schemas/util.sql: Added force_update_view()
- 04:23 AM Revision 8320: bin/make_analytical_db: Commented out export_analytical_db because we are not yet using the analytical DB in MySQL, and it doesn't make sense to generate a large, unused CSV export each time
- 04:19 AM Revision 8319: bin/export_analytical_db: Replaced analytical_aggregate with analytical_stem
- 03:53 AM Revision 8318: inputs/FIA/occurrence_all/: Updated header.csv for new column order
- 03:40 AM Revision 8317: inputs/FIA/occurrence_all/import: Use directional joins (LEFT/RIGHT JOIN) instead of inner joins to ensure that the PostgreSQL query planner always joins starting with the TREE table. Note that the directional joins are now needed for a different reason than when they were initially added, which had been to avoid slow sorts. The sorts (at least for LIMIT-only queries) went away when small tables such as COUNTY and REF_UNIT were added to the joins.
- 01:16 AM Revision 8316: inputs/FIA/*/map.csv: Changed newlines between table and field name to - because the newlines mess up the flow of queries and also break pgAdmin's display of EXPLAIN output. The - was chosen because it's a non-whitespace character that linewraps in browsers, phpPgAdmin, and Google spreadsheets (although unfortunately not in pgAdmin). It is better than space because you can set a text editor to treat it as a word character, allowing the entire column name (<table>-<field>) to be selected by double-clicking it.
04/03/2013
- 09:55 PM Revision 8315: Added planning/workflow/normalized_vs_denormalized/denormalized.generic_standardizations.png (a slide from Brad's bien3_architecture_denormalized.pptx PowerPoint), which shows the staging table preprocessing particularly well
- 09:45 PM Revision 8314: README.TXT: Full database import: record the import times in inputs/import.stats.xls: Added instructions for what to do if the rightmost imports start getting truncated due to the 255-column limit in spreadsheets. (This will occur in 8 imports.)
- 09:32 PM Revision 8313: inputs/import.stats.xls: Removed the previous imports from the current tab because they are also in the 2012-6~9 tab, and should not be in two places
- 09:28 PM Revision 8312: inputs/import.stats.xls: Updated import times. MO and FIA have been refreshed.
04/02/2013
- 04:17 PM Revision 8311: Removed no longer needed inputs/GBIF/import. Use ./run instead.
- 04:17 PM Revision 8310: Removed no longer needed inputs/GBIF/_MySQL/import. Use ./run instead.
- 04:16 PM Revision 8309: inputs/GBIF/_MySQL/run: import: Run make directly instead of via ./import
- 04:15 PM Revision 8308: inputs/GBIF/_MySQL/run: Use new import.run, which defines all()
- 04:06 PM Revision 8307: Added planning/workflow/normalized_vs_denormalized/bien3_architecture_(de)normalized.pptx
- 03:57 PM Revision 8306: Added planning/workflow/normalized_vs_denormalized/BIEN-modArch-Dec2010 NS-SBD 1.4.ppt.url
- 03:50 PM Revision 8305: planning/workflow/: Moved normalized vs. denormalized files to separate normalized_vs_denormalized/ subfolder
- 03:21 PM Revision 8304: Regenerated inputs/ACAD/Specimen/logs/steps.by_col.log.sql
- 03:15 PM Revision 8303: inputs/GBIF/raw_occurrence_record/run: Override MySQL_export() so $filter can be customized
- 03:13 PM Revision 8302: inputs/GBIF/table.run: import(): Updated for lib/table.run template changes
- 03:09 PM Revision 8301: lib/table.run: template: import(): Also pass "$@" to superclass method
- 03:08 PM Revision 8300: lib/table.run: template: Use "$FUNCNAME" instead of hardcoding import
- 03:02 PM Revision 8299: Added inputs/GBIF/MySQL_export, used by ./table.run
- 02:57 PM Revision 8298: lib/util.run: echo_func: Fixed bug where need to use BASH_LINENO[0] for the line #s to match up with the files. For some reason the required array indexes for BASH_SOURCE (1) and BASH_LINENO (0) differ by one.
- 02:51 PM Revision 8297: inputs/GBIF/run: Use new import.run, which defines all()
- 02:51 PM Revision 8296: lib/table.run: Use new import.run, which defines all()
- 02:49 PM Revision 8295: Added lib/import.run
- 02:48 PM Revision 8294: lib/util.run: echo_func: Include the line # of the function to make it easier to find where the code being run is
- 02:32 PM Revision 8293: lib/table.run: Added all (default target)
- 02:26 PM Revision 8292: lib/util.run: run_cmd: If bash exited with an error, don't run the "$@" command. This test is necessary because `trap run_cmd EXIT` will run run_cmd as the result of *any* exit from the shell, including an error.
- 02:21 PM Revision 8291: *run: Use -e option to bash on the #! line instead of separate `set -o errexit` line so that there is no issue with the `set -o errexit` line getting separated from the #! line (errexit is required for the scripts to work properly)
- 02:09 PM Revision 8290: lib/util.run: run_cmd: When no command specified, default to running the `all` command, just like make
- 02:07 PM Revision 8289: lib/util.run: Run run_cmd at shell exit (using trap) instead of requiring every runscript to have `run_cmd ` at the end of it
- 01:49 PM Revision 8288: Added inputs/GBIF/run
- 01:48 PM Revision 8287: Added inputs/GBIF/raw_occurrence_record/run
- 01:47 PM Revision 8286: Added inputs/GBIF/table.run
- 01:45 PM Revision 8285: Added inputs/GBIF/_MySQL/run
- 01:42 PM Revision 8284: lib/util.run: fwd: Check that $subdirs is defined. Added $subdirs to usage.
- 01:39 PM Revision 8283: lib/util.run: fwd: Added usage
- 01:32 PM Revision 8282: lib/table.run: Switched from echo_run to echo_func
- 01:16 PM Revision 8281: lib/util.run: run_cmd: Echo the command being run, including the top-level run script. This is in addition to the echoing of the command in the function itself (using echo_func), which provides *both* the runscript that was run *and* the file where the invoked command was actually located (which may be different due to includes).
- 01:12 PM Revision 8280: lib/util.run: Echo the command at the beginning of each function using new echo_func, instead of having to type echo_run before every call to a function. Note that because echo_func uses BASH_SOURCE, the path to the file containing the function will be included in the debug message, which greatly facilitates locating which file a command is in.
- 01:08 PM Revision 8279: lib/util.run: Added echo_func
- 12:50 PM Revision 8278: lib/util.run: Added echo_cmd and use it in echo_run
- 12:46 PM Revision 8277: lib/util.run: echo_cmd(): Renamed to echo_run for clarity, because it also runs the command
- 12:39 PM Revision 8276: lib/util.run: Added inline_make()
- 12:39 PM Revision 8275: lib/util.run: Added echo_stdin()
- 12:30 PM Revision 8274: bin/my2pg_export: Put --password first because it's an authentication-related option
- 10:52 AM Revision 8273: Added lib/table.run, which includes the commands in import.sh but uses run scripts to allow running commands other than just import. (For example, map_table or postprocess can be run separately. Uninstall-related commands which would not belong in an import script can also be added, because import is only one of many commands a run script can offer.)
- 10:35 AM Revision 8272: Added lib/util.run with general functions and template for run scripts (a bash-based replacement for make). Unlike make, run scripts support full bash functionality including multiline commands. The run script template also includes syntax for various kinds of relative includes in bash.
- 12:03 AM Revision 8271: lib/common.Makefile: Added $(require_var)
04/01/2013
- 10:42 PM Revision 8270: bin/publish_analytical_db: Fixed bug where need to remove `ESCAPED BY '"'` because this would causing " followed by an escape sequence char to be interpreted specially (e.g. "n -> \n). MySQL automatically takes care of quote doubling when you specify `FIELDS OPTIONALLY ENCLOSED BY`.
- 10:13 PM Revision 8269: lib/common.Makefile: Compression: Added `%:: %.gz`, `%.gz: %`
- 08:07 PM Revision 8268: planning/workflow/import_process_comparison.odg: Moved "staging tables" under the method labels to reduce empty space
- 07:52 PM Revision 8267: planning/workflow/import_process_comparison.odg: Removed margins so the labels would align with the page margin on the Import process wiki page <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Import_process>
- 07:32 PM Revision 8266: Added planning/workflow/import_process_comparison.odg and .png export
- 06:12 PM Revision 8265: lib/db_xml.py: put_table(): Fixed bug where command to advance start to fetch next set was unintentionally deleted when removing the is_view check
- 06:11 PM Revision 8264: inputs/UNCC/Specimen/new_terms.csv: Updated for updated VegCore vocab
- 03:53 PM Revision 8263: inputs/GBIF/_MySQL/GBIFPortalDB-2013-02-20.data.sql.md5: Regenerated after appending agent table to GBIFPortalDB-2013-02-20.data.sql
- 03:51 PM Revision 8262: Added inputs/GBIF/_MySQL/GBIFPortalDB-2013-02-20.data.sql.gz.md5
03/28/2013
- 08:16 PM Revision 8261: Added inputs/GBIF/raw_occurrence_record/ from refresh
- 08:07 PM Revision 8260: inputs/GBIF/MySQL.schema.sql: Regenerated with inline enum type translated to CHECK constraint
- 08:07 PM Revision 8259: bin/my2pg: Translate inline enum type to CHECK constraint
- 07:43 PM Revision 8258: Added inputs/GBIF/**/MySQL.schema.sql
- 07:42 PM Revision 8257: Added inputs/GBIF/_MySQL/MySQL.*.sql.make
- 07:36 PM Revision 8256: inputs/FIA/: Archived no longer used subdirs from BIEN2 export
- 07:29 PM Revision 8255: inputs/FIA/: Archived no longer used subdirs from BIEN2 export
- 07:22 PM Revision 8254: inputs/input.Makefile: SVN: add: Removed Source/map.csv prerequisite because it is not related to adding unversioned files in the dir. It was originally a prerequisite in order to auto-create it when the datasource dir is first created, but the map.csv recipe does not currently create metadata-only map.csvs. In the future, metadata-only map.csvs will be replaced with constant columns added to the applicable tables.
- 07:19 PM Revision 8253: Added inputs/FIA/_archive
- 07:19 PM Revision 8252: inputs/input.Makefile: %/map.csv: Fixed bug where can only make header.csv if map.csv does not exist, because some subdirs are metadata-only and don't have a corresponding DB table
- 07:02 PM Revision 8251: README.TXT: Datasource setup: Install the staging tables: For a MySQL .sql export: Documented which password to use at each of the two password prompts my2pg_export will give you. You could also embed the value of the 2nd prompt in the _MySQL/*.make file using `--password="$(cat path/to/config/bien_password)"`.
- 06:56 PM Revision 8250: README.TXT: Datasource setup: Install the staging tables: Removed requirement that `make inputs/<datasrc>/reinstall quiet=1 &` be run on vegbiendev for MySQL .sql exports, because the hostname is now set to vegbiendev instead of localhost
- 06:38 PM Revision 8249: inputs/input.Makefile: sql/install: Use psql_script_vegbien instead of $(psqlNoSearchPath) (which uses psql_verbose_vegbien) because the insert statement for each data row should not be echoed
- 06:14 PM Revision 8248: inputs/FIA/occurrence_all/import: Run remake_VegBIEN_mappings at end to keep mappings to next stage of import process up to date
- 06:14 PM Revision 8247: inputs/FIA/occurrence_all/: Accepted new test output
- 06:13 PM Revision 8246: lib/import.sh: remake_VegBIEN_mappings(): Also remake VegBIEN.csv and test.xml.ref use `make test`
- 06:11 PM Revision 8245: lib/import.sh: Added remake_VegBIEN_mappings()
- 06:10 PM Revision 8244: inputs/input.Makefile: %/map.csv: make $*/header.csv first in case it doesn't exist (e.g. if it has been deleted so that it will be remade)
- 06:07 PM Revision 8243: inputs/FIA/occurrence_all/map.csv: Regenerated using new input table mappings
- 05:47 PM Revision 8242: lib/import.sh: Added make() and use it instead of the full make command
- 05:23 PM Revision 8241: inputs/input.Makefile: postprocess: Use %/postprocess instead of %/postprocess.sql/run so $*/import is also run
- 05:21 PM Revision 8240: inputs/FIA/: Ran inputs/FIA/import. This maps to VegCore's commonName.
- 05:19 PM Revision 8239: inputs/input.Makefile: %/postprocess: Also run the $*/import script, if it exists. Note that this is not the same as the %/import make target.
- 05:12 PM Revision 8238: inputs/input.Makefile: %/postprocess.sql/run: Factored out into separate %/postprocess command, which can eventually also perform other actions
- 04:59 PM Revision 8237: inputs/FIA/PLOT/map.csv: ELEV: Remapped to elevation_ft, assuming units based on the actual elevation of the region for a sample plot record
- 04:27 PM Revision 8236: inputs/VegBank/taxonobservation_/map.csv: Mapped int_currplantcommon to vernacularName
- 04:25 PM Revision 8235: mappings/VegCore.htm: Renamed salvias_plots table plotMetadata to PlotMetadata because of SALVIAS refresh on nimoy
- 04:18 PM Revision 8234: mappings/VegCore.htm: Regenerated from wiki. Added flower, fruit, commonName.
- 03:37 PM Revision 8233: mappings/Makefile: $(vocab); bin/redmine_synonyms: Support crossed out (deprecated) terms
- 03:24 PM Revision 8232: README.TXT: Maintenance: VegCore data dictionary: Added steps to update the data dictionary's Tables section if necessary
- 02:14 PM Revision 8231: inputs/GBIF/_MySQL/Makefile: %.data.sql: Added agent table
- 01:18 PM Revision 8230: Added inputs/GBIF/_MySQL/GBIFPortalDB-2013-02-20.data.sql.md5
- 01:11 PM Revision 8229: Added inputs/GBIF/_MySQL/GBIFPortalDB-2013-02-20.schema.sql
- 11:02 AM Revision 8228: Added web/main/svn*/, now using .htaccess to forward to Redmine/*
- 10:55 AM Revision 8227: Removed web/main/svn, svn-web symlinks because they need to be .htaccess-es in order for the relative mod_rewrite commands to work correctly
- 10:50 AM Revision 8226: Added web/main/svn, svn-web symlinks to Redmine/* for shorter URLs
- 10:49 AM Revision 8225: Added web/main/Redmine/svn-web/
- 08:28 AM Revision 8224: inputs/GBIF/: Added scripts for subsetting refresh
- 12:24 AM Revision 8223: lib/sql.py: table_order_by(): Documented that it returns None if table is a view, because table_cluster_on() would return None. This is necessary for inputs/FIA/occurrence_all/ sorting to work correctly, because specifying a manual sort order would prevent the query planner from just using fast nested loop joins and instead cause it to perform a slow sort. (This appears to be a bug in the query planner, because when the column list specified matches the joined-on indexes, there should be no need for post-nested loop re-sorting.)
- 12:20 AM Revision 8222: inputs/FIA/occurrence_all/test.xml.ref: Updated inserted row count for new row sort order
- 12:19 AM Revision 8221: lib/db_xml.py: put_table(): Fixed bug where also need to advance start to fetch next set when table is a view, because the views that are now being used with the import (inputs/FIA/occurrence_all/) are static rather than dynamic and do not return different rows after the previous set of rows has been imported
03/27/2013
- 11:43 PM Revision 8220: inputs/FIA/occurrence_all/import: Removed no longer applicable comment that directional joins are needed for PostgreSQL query planner to avoid slow sorts
- 11:40 PM Revision 8219: inputs/FIA/TREE/import: Reclustered table by TREE.parent path index, to facilitate path-order joins
- 11:39 PM Revision 8218: inputs/FIA/occurrence_all/import: Changed all RIGHT JOINs to inner joins so that tables would be joined in path order (i.e. general->specific). This optimizes the incremental joins so that the small tables are joined to each other before being joined to the large tables, rather than each row of the large tables being looked up in the small tables. This effect may not be noticeable for small LIMIT values, but would become apparent for large LIMIT values, such as the 1-million-row partitions used by db_xml.put_table() for column-based import. Note that inner joins used to cause the query planner to produce incorrect results containing slow sorts, but now this appears to no longer be an issue, perhaps because the result is not sorted by the TREE.ID index (which is not in the same order as the path indexes *.unique, *.parent).
- 10:46 PM Revision 8217: inputs/FIA/occurrence_all/import: Removed trailing whitespace
- 10:30 PM Revision 8216: Removed unused inputs/FIA/COND_unique/. Use COND instead.
- 09:52 PM Revision 8215: inputs/FIA/import: Use `set -o errexit` instead of putting ` || exit` after each command
- 09:52 PM Revision 8214: lib/import.sh: map_table(): Removed unneeded () around psql. This also fixes a bug where an error exit status from psql would not have aborted the script because `set -o errexit` does not apply to commands enclosed in (). For () you need to use ` || exit` instead (or ` || return` inside a function).
- 09:42 PM Revision 8213: lib/import.sh: Use `set -o errexit` so any command that exits with an error aborts the script. Note that a command's exit status can still be ignored using ` || true`. Removed no longer needed ` || return` in functions.
- 09:40 PM Revision 8212: schemas/util.sql: Renamed rename_if_exists() to try_create() because it can be used to create a column in any way, not just by renaming another column
- 09:33 PM Revision 8211: lib/import.sh: functions: abort if a command encounters an error
- 09:17 PM Revision 8210: schemas/VegCore/mk_derived: Added cultivated from oldGrowth
- 09:16 PM Revision 8209: schemas/util.sql: Added try_mk_derived_col()
- 08:35 PM Revision 8208: inputs/FIA/*/import: Run mk_derived after postprocessing commands
- 08:28 PM Revision 8207: inputs/FIA/import_order.txt: Added occurrence_all/
- 08:23 PM Revision 8206: mappings/VegCore-VegBIEN.csv: subplotID,subplot -> location.sourceaccessioncode: Fixed bug where need /_first to handle the case where both subplotID and subplot are provided
- 08:15 PM Revision 8205: Added inputs/FIA/map.csv, which maps shared columns to VegCore
- 08:12 PM Revision 8204: inputs/FIA/FIA_COND_unique/test.xml.ref: Updated now that PLOT, CONDID have been mapped
- 08:12 PM Revision 8203: inputs/FIA/*/map.csv for pre-refresh tables: Added back * before unmapped column names
- 08:03 PM Revision 8202: lib/csvs.py: stream_info(): Fixed bug where headers with multiline columns were not supported because only the first line (not the first multiline row) is sniffed for the dialect
- 06:56 PM Revision 8201: inputs/input.Makefile: %/header.csv: Fixed bug where newlines inside column names were incorrectly formatted by psql's table header formatting, by using COPY TO STDOUT instead
- 05:28 PM Revision 8200: schemas/util.sql: Added do_optionally_ignore()
- 04:28 PM Revision 8199: schemas/util.sql: Renamed rename_if_exists() to try_create() because it can be used to create a column in any way, not just by renaming another column
- 04:12 PM Revision 8198: lib/import.sh: Added mk_derived(). Added mk_derived to usage template.
- 04:11 PM Revision 8197: Added schemas/VegCore/mk_derived, which will be run in the import scripts
- 04:09 PM Revision 8196: lib/import.sh: psql(): Set psql vars :schema, :table, :table_str for use by the psql commands
- 03:22 PM Revision 8195: lib/import.sh: Export $schema, $table so they are available to programs invoked within an import script, which should not reset these vars if they include import.sh
- 03:20 PM Revision 8194: lib/import.sh: Only set $table, $schema if they don't already exist
- 03:11 PM Revision 8193: lib/import.sh: Added $root_dir and use it in $bin_dir
- 03:11 PM Revision 8192: inputs/FIA/*/import: Use new mk_*_col()
- 02:50 PM Revision 8191: schemas/*functions.sql: Renamed to *util.sql because now that these schemas are used by the new-style import scripts, there can be more than just functions in them
- 02:43 PM Revision 8190: schemas/util.sql: Added mk_const_col()
- 02:37 PM Revision 8189: schemas/util.sql: Added type_qual()
- 02:34 PM Revision 8188: schemas/util.sql: mk_derived_col(): Added "idempotent" comment
- 02:23 PM Revision 8187: schemas/util.sql: Added mk_derived_col()
- 02:22 PM Revision 8186: inputs/FIA/COND/import: oldGrowth: Updated expr column names
- 01:49 PM Revision 8185: schemas/util.sql: Added typeof(text, regtype)
- 12:54 PM Revision 8184: inputs/FIA/*/import: Removed util. before function names because util is in the search_path
- 12:43 PM Revision 8183: schemas/*functions.sql: Renamed to *util.sql because now that these schemas are used by the new-style import scripts, there can be more than just functions in them
03/25/2013
- 11:19 PM Revision 8182: schemas/functions.sql: Added existing_cols()
- 11:12 PM Revision 8181: schemas/functions.sql: col_type(): Fixed bug where a NULL col name crashed the undefined_column throw, because MESSAGE can't be NULL and the NULL name was nulling out the entire message
- 11:08 PM Revision 8180: schemas/functions.sql: Added col_exists()
- 10:31 PM Revision 8179: inputs/FIA/COND/map.csv: Mapped SLOPE, ASPECT
- 10:23 PM Revision 8178: web/main/.htaccess: remove linewraps (of the form table.path.vg/_-term) used to create a newline for Google spreadsheets
- 09:45 PM Revision 8177: inputs/FIA/*/map.csv: Replaced . between table and column name with newline, so that table viewers like pgAdmin will display both the table and column name at the left edge of the header cell, rather than displaying only the table name because the column name doesn't fit. This fixes the problem of seeing a bunch of columns whose names all start with a table name, and not knowing what each of them is. It also preserves the ability to see at a glance which table a column is in, which helps in navigating wide tables. Removed * before unmapped terms, because whether a term is mapped is generally obvious from the table name itself.
- 09:01 PM Revision 8176: inputs/input.Makefile: %/.map.csv.last_cleanup: Run fix_line_endings after canon/translate to standardize Python's \r\n line endings back to \n. This prevents issues with mixed line endings because LibreOffice (and probably Excel) treat all cell-internal line endings as \n but row line endings as whatever the file had, while text editors like jEdit translate all line endings to whatever the autodetected line ending is. (This creates spurious line ending diffs when a map spreadsheet containing multiline cells is edited in a text editor.)
- 08:45 PM Revision 8175: Added bin/fix_line_endings to standardize \r\n line endings to \n
- 08:12 PM Revision 8174: inputs/FIA/COND/import: Renamed COND.oldgrowth to VegCore name oldGrowth
- 07:52 PM Revision 8173: inputs/FIA/*/map.csv: Ensured that joined columns are globally unique, so they don't map to an ambiguous VegCore term in the future
- 07:38 PM Revision 8172: inputs/FIA/*/map.csv: Mapped terms to VegCore
- 07:22 PM Revision 8171: schemas/functions.sql: col_type(): Include column name in error message
- 06:57 PM Revision 8170: inputs/FIA/*/import: Updated column names to match map.csv
- 06:47 PM Revision 8169: schemas/functions.sql: col_type(): Raise undefined_column exception if column does not exist, instead of silently returning NULL
- 06:34 PM Revision 8168: inputs/FIA/import: Abort if any invoked script encounters an error
- 05:44 PM Revision 8167: planning/timeline/timeline.2013.xls: Updated for current progress
- 04:55 PM Revision 8166: inputs/FIA/*/map.csv: Removed no longer needed leading . from joined fields (globally-unique terms), because functions.to_global_col_names() is not used anymore
- 04:46 PM Revision 8165: Added inputs/FIA/occurrence_all/, which combines all the core tables in a denormalized view. Note that it is not necessary to materialize this view into a (large) denormalized table, because the unique indexes and left/right joins allow the rows to be denormalized on the fly.
- 04:36 PM Revision 8164: inputs/FIA/*/import: Use map_table to set column names based on the contents of map.csv, instead of using functions.to_global_col_names() and functions.rename_if_exists(). Added map.csv for all tables.
- 03:19 PM Revision 8163: inputs/FIA/: Changed postprocess.sql scripts to import scripts that can be run directly. Added top-level inputs/FIA/import to run all of them together.
- 03:05 PM Revision 8162: inputs/FIA/COND/postprocess.sql: Removed trailing whitespace
- 02:25 PM Revision 8161: Added lib/import.sh, for use by new, simpler import scripts used by FIA. Note that for now, input.Makefile is still used to create map.csv.
03/22/2013
- 11:13 PM Revision 8160: inputs/input.Makefile: Moved postprocess.sql from $(exportHeader) to %/install because that is not part of the $(exportHeader) functionality. Added %/header.csv and use it in $(exportHeader).
- 11:05 PM Revision 8159: inputs/input.Makefile: $(catSrcs): Fixed bug where need to use $(nonHeaderSrcs) instead of $(srcs) to exclude header.csv
- 08:07 PM Revision 8158: schemas/functions.sql: map: Added additional columns that are present in the standard map spreadsheet format (filter, notes). These columns are necessary to make COPY FROM work, because it requires the # of columns to be the same in the input data and the output table.
- 07:39 PM Revision 8157: inputs/input.Makefile: Moved $(cleanup) from $(exportHeader) to %/install because this is not part of exportHeader's functionality
- 07:29 PM Revision 8156: inputs/input.Makefile: $(mkSrcMap): Use header.csv instead of the header of the CSVs, so that the column list in the map spreadsheet matches the actual DB table
- 07:18 PM Revision 8155: inputs/input.Makefile: %.sql/run: Change to the directory the file is located in, so that includes (\i) are relative to the file, rather than relative to whatever happens to be the current directory
- 07:15 PM Revision 8154: inputs/input.Makefile: %/install: Always generate a header.csv, even for CSV inputs with their own header. This will include the *actual* column names in the staging table, which may differ from their names in the CSVs (e.g. the addition of row_num). Note that header.csv is not included in the CSVs list itself, and will not override the header or dialect in them.
- 06:09 PM Revision 8153: schemas/functions.sql: Added set_col_names()
- 06:08 PM Revision 8152: schemas/functions.sql: rename_if_exists(): Also ignore duplicate_column exceptions, which are generated when a column is renamed to itself (as well as when two columns are renamed to the same place)
- 06:02 PM Revision 8151: schemas/functions.sql: Added col_names(regclass), which unlike col_names(regtype) returns names in the order they are in the table
- 04:56 PM Revision 8150: schemas/functions.sql: Added map_values()
- 02:50 PM Revision 8149: schemas/functions.sql: map_get(): Fixed bug where can't use STRICT in EXECUTE INTO because there will sometimes be no match, causing a "query returned no rows" error
- 02:33 PM Revision 8148: schemas/functions.sql: rename_cols(): Support any renames type with an -> operator
- 02:27 PM Revision 8147: schemas/functions.sql: Added operator ->(regclass, text)
- 01:49 PM Revision 8146: schemas/functions.sql: Added map_get()
- 01:38 PM Revision 8145: schemas/functions.sql: table2hstore(): Made it STABLE instead of IMMUTABLE because the input table is not constant
- 01:36 PM Revision 8144: schemas/functions.sql: Added table2hstore()
- 01:34 PM Revision 8143: schemas/functions.sql: Added reset_map_table()
- 12:51 PM Revision 8142: schemas/functions.sql: Added truncate()
- 12:28 PM Revision 8141: schemas/functions.sql: mk_map_table(): Use the sql language instead of plpgsql because EXECUTE is not used directly, so plpgsql is not actually needed
- 12:25 PM Revision 8140: schemas/functions.sql: mk_map_table(): Store map table schema in separate `map` table and extend it using LIKE, for easier maintainability of the map schema
03/21/2013
- 08:27 AM Revision 8139: schemas/functions.sql: Added mk_map_table()
- 07:54 AM Revision 8138: schemas/functions.sql: ensure_prefix(): Made it IMMUTABLE instead of STABLE
- 07:37 AM Revision 8137: schemas/functions.sql: Added rename_cols()
- 06:01 AM Revision 8136: inputs/FIA/*/postprocess.sql: Avoid using :table, :table_str so that the commands in the script can also be run by pasting them into pgAdmin
- 02:13 AM Revision 8135: README.TXT: Full database import: Manual steps to run TNRS/remake analytical DB: Added `export version=<version>` to ensure that the import is run into the correct schema. Since these instructions are for running commands separately from the rest of the import, it's important to first ensure that the import environment is set up properly.
- 02:08 AM Revision 8134: schemas/vegbien.ERD.mwb: Added taxon_trait to ERD
- 02:04 AM Revision 8133: schemas/vegbien.ERD.mwb: Regenerated exports
- 01:58 AM Revision 8132: schemas/vegbien.sql: Removed unused analytical_aggregate table, because analytical_stem provides much more detailed, higher-quality data, both in terms of the number or of rows and the number of columns. analytical_aggregate has also long been out of sync with the analytical DB schema, and it doesn't make sense to spend processing time in make_analytical_db to perform the DISTINCT ON if the table isn't being used. We may revisit analytical_aggregate later once we have ID fields for each entity in the DISTINCT ON and can avoid DISTINCTing on all analytical_aggregate columns.
- 01:53 AM Revision 8131: schemas/vegbien.sql: Removed unused analytical_aggregate table, because analytical_stem provides much more detailed, higher-quality data, both in terms of the number or of rows and the number of columns. analytical_aggregate has also long been out of sync with the analytical DB schema, and it doesn't make sense to spend processing time in make_analytical_db to perform the DISTINCT ON if the table isn't being used. We may revisit analytical_aggregate later once we have ID fields for each entity in the DISTINCT ON and can avoid DISTINCTing on all analytical_aggregate columns.
- 01:27 AM Revision 8130: inputs/FIA/*/postprocess.sql: Added index on *.CN (autogen IDs)
- 01:25 AM Revision 8129: README.TXT: Full database import: Added steps to use `screen` to allow recovering from a closed terminal window
- 01:08 AM Revision 8128: inputs/FIA/TREE/postprocess.sql: TREE.unique index: Renamed to TREE.ID because this is on an autogenerated pkey rather than on domain values (for which a set of unique columns has not yet been found and may not exist)
- 01:03 AM Revision 8127: inputs/FIA/REF_SPECIES/postprocess.sql: Matched SPECIES_SYMBOL to .SYMBOL. Added .SYMBOL_TYPE for use in joining to REF_PLANT_DICTIONARY.
- 12:41 AM Revision 8126: Added inputs/FIA/REF_UNIT/postprocess.sql
- 12:36 AM Revision 8125: Added inputs/FIA/REF_RESEARCH_STATION/postprocess.sql
- 12:17 AM Revision 8124: Added inputs/FIA/COUNTY/postprocess.sql
- 12:02 AM Revision 8123: Added inputs/FIA/REF_PLANT_DICTIONARY/postprocess.sql
03/20/2013
- 05:16 PM Revision 8122: inputs/FIA/COND/postprocess.sql: Matched COND.HABTYPCD1, COND.HABTYPCD1_PUB_CD to REF_HABTYP_DESCRIPTION
- 05:07 PM Revision 8121: inputs/input.Makefile: Staging tables installation: $(exportHeader): Fixed bug where need to run postprocess.sql before exporting the header, because *it* can change the column names
- 05:02 PM Revision 8120: inputs/input.Makefile: Staging tables installation: $(exportHeader): export the header before running $(cleanup), because the header is not affected by the data cleanup operations and thus can be generated right away, to allow mapping while the cleanup operations run
- 04:30 PM Revision 8119: inputs/FIA/REF_HABTYP_DESCRIPTION/postprocess.sql: Prepare columns for joining with COND
- 03:23 PM Revision 8118: inputs/input.Makefile: Staging tables installation: $(exportHeader): Fixed bug where need to use psql_script_vegbien instead of the psql_verbose_vegbien used by $(psqlAsBien), to avoid echoing commands as part of the exported header
- 03:10 PM Revision 8117: Added planning/workflow/(de)normalized_import.mappings.png
- 03:04 PM Revision 8116: Added planning/workflow/denormalized_import.png, normalized_import.png
- 10:37 AM Revision 8115: web/main/IH/: Added lowercase alias
- 10:32 AM Revision 8114: Added web/main/IH/
- 10:12 AM Revision 8113: inputs/input.Makefile: Staging tables installation: Added postprocess target, which runs all the postprocess.sql files
- 09:34 AM Revision 8112: inputs/FIA/REF_SPECIES/postprocess.sql: Cast ID column to integer
- 08:52 AM Revision 8111: inputs/FIA/*/postprocess.sql: Cluster tables by their *.unique index for faster joins
- 08:51 AM Revision 8110: inputs/FIA/*/postprocess.sql: Cast ID columns to integer using new functions.set_col_types()
- 08:49 AM Revision 8109: bin/psql_verbose_vegbien: Run with client_min_messages = NOTICE to display notices for debugging. This is supposed to be the default, but apparently isn't.
- 08:47 AM Revision 8108: inputs/input.Makefile: BIEN commands: $(psqlAsBien): Use psql_verbose_vegbien instead of psql_script_vegbien so that timings and notices are displayed, which is useful for profiling and debugging
- 08:32 AM Revision 8107: schemas/functions.sql: Added col_cast and set_col_types()
- 07:45 AM Revision 8106: schemas/functions.sql: Added col_ref, col_type()
- 06:51 AM Revision 8105: schemas/functions.sql: Added cluster_once()
- 06:36 AM Revision 8104: schemas/functions.sql: Added cluster_index()
- 05:55 AM Revision 8103: schemas/functions.sql: create_if_not_exists(): Also handle duplicate_column exceptions
- 05:54 AM Revision 8102: schemas/functions.sql: Added rename_if_exists()
- 05:48 AM Revision 8101: inputs/FIA/COND/postprocess.sql: Renamed oldgrowth to COND.oldgrowth so it wouldn't be renamed by to_global_col_names()
- 04:28 AM Revision 8100: inputs/FIA/COND/postprocess.sql: Added oldgrowth column as part of the postprocessing instead of as part of the view that left joins the core tables together. This avoids needing to regenerate the oldgrowth field whenever the view is queried or materialized.
- 04:01 AM Revision 8099: inputs/FIA/TREE/postprocess.sql: Added index on columns that join to parent tables
- 03:00 AM Revision 8098: inputs/FIA/*/postprocess.sql: Removed table prefix from globally-unique columns that should be joined on
- 02:25 AM Revision 8097: schemas/functions.sql: Marked STRICT functions as such
- 02:22 AM Revision 8096: schemas/functions.sql: col_global_names(): Treat any column name that contains . as already being globally unique, and don't prepend the table name. This allows renaming the table columns after running col_global_names(), without causing the table name to be re-prepended the next time col_global_names() is run.
- 02:09 AM Revision 8095: schemas/functions.sql: Added contains()
- 02:07 AM Revision 8094: schemas/functions.sql: Added create_if_not_exists()
- 01:28 AM Revision 8093: inputs/FIA/*/postprocess.sql: Use functions.to_global_col_names() to ensure that all column names are globally unique. This makes it easy to join the tables together without worrying about column name collisions.
- 01:15 AM Revision 8092: inputs/FIA/*/postprocess.sql: Use new functions.create_if_not_exists() to allow re-running postprocess.sql idempotently
03/19/2013
- 11:48 PM Revision 8091: inputs/input.Makefile: Staging tables installation: %/install: Use new %.sql/run to run postprocess.sql
- 11:47 PM Revision 8090: inputs/input.Makefile: Staging tables installation: Added %.sql/run to run postprocess.sql, etc. separately from the install targets they are a part of
- 11:47 PM Revision 8089: inputs/input.Makefile: Staging tables installation: Added %.sql/run to run postprocess.sql, etc. separately from the install targets they are a part of
- 10:43 PM Revision 8088: schemas/functions.sql: Added to_global_col_names()
- 10:22 PM Revision 8087: schemas/functions.sql: col_global_names(): Use new functions.ensure_prefix() to only add the table name prefix if it doesn't already exist. This makes the function idempotent.
- 10:19 PM Revision 8086: schemas/functions.sql: Added ensure_prefix()
- 10:17 PM Revision 8085: schemas/functions.sql: Added has_prefix()
- 10:09 PM Revision 8084: schemas/functions.sql: Added col_global_names()
- 09:59 PM Revision 8083: schemas/functions.sql: Added name(regtype)
- 09:43 PM Revision 8082: schemas/functions.sql: Added col_names()
- 09:27 PM Revision 8081: root Makefile: Installation: Fixed bug where need to run schemas/public/install separately because schemas/install installs only the util schemas
- 09:26 PM Revision 8080: root Makefile: Installation: install util schemas (temp functions py_functions) before inputs, so that inputs can use util functions in their postprocess.sql or create.sql scripts. (However, they must not use util functions in views or index functions, because these would be cascadingly deleted whenever the util schemas are reinstalled before an import.)
- 08:07 PM Revision 8079: README.TXT: Single datasource import: Added by_col=1 to all commands
- 02:28 AM Revision 8078: mappings/VegCore-VegBIEN.csv: locationRemarks: Remapped to locationnarrative because location.notespublic is a boolean field
- 02:05 AM Revision 8077: lib/sql_io.py: mk_errors_table(): Create a unique index on the MD5 of the value and error instead of on the values directly, because some strings are too long to index (e.g. row 2537268 of MO.Specimen causes an error "index row size 3032 exceeds maximum 2712 for index [...] Values larger than 1/3 of a buffer page cannot be indexed")
- 12:49 AM Revision 8076: inputs/import.stats.xls: Updated import times
03/16/2013
- 02:16 PM Revision 8075: bin/map: No mappings warning: Added explanation that this could also be due to no column name matches, and hint to check if you are importing the correct input table
- 01:45 PM Revision 8074: inputs/MO/: Renamed Specimen.2/ -> now available Specimen/
- 01:42 PM Revision 8073: inputs/MO/: Removed old import in Specimen/
- 01:33 PM Revision 8072: Refreshed MO
- 12:44 PM Revision 8071: csvs.py: TsvReader.next(): Fixed bug where empty line needs to be separately returned as [], because csv.reader would interpret it as EOF since the line ending has already been removed
- 12:25 PM Revision 8070: csvs.py: sniff(): TSVs: Turn off quoting because TSVs use \-escapes instead of quotes to escape delimeters, newlines, etc.
- 11:49 AM Revision 8069: csvs.py: InputRewriter.readline(): Surround function in a try block that prints all exceptions, so that debugging information is available if an error occurs when this stream is used as input for psycopg's copy_expert() (COPY FROM)
- 06:56 AM Revision 8068: Populated inputs/MO/import_order.txt
- 06:46 AM Revision 8067: Refreshed SALVIAS
- 06:33 AM Revision 8066: Added web/main/CTFS/
- 06:21 AM Revision 8065: inputs/SALVIAS/: Regenerated salvias_*.schema.sql from the MySQL version, to take advantage of my2pg improvements. The placeholder *_index columns which take the place of MySQL's inline index definitions have now been replaced by no-op CHECK constraints, so that there are no longer lots of dummy *_index columns in the map spreadsheets.
- 05:52 AM Revision 8064: Added web/main/Redmine/ alias to VegBIEN/Redmine/
- 05:51 AM Revision 8063: Added web/main/VegBIEN/Redmine/
- 05:48 AM Revision 8062: web/main/VegBIEN/.htaccess: Forward to new db/ subdir
- 05:47 AM Revision 8061: Added web/main/VegBIEN/db/
- 05:45 AM Revision 8060: web/main/**/.htaccess: Removed RewriteCond -l tests because one of the -d or -f tests will always also pass, making the -l test unnecessary
- 05:38 AM Revision 8059: web/main.conf: Added tolower RewriteMap
- 05:19 AM Revision 8058: web/main/.htaccess: use separate lowercase version when available: Also support input strings in mixed case which is not the default capitalization, in addition to all-lowercase strings
- 05:18 AM Revision 8057: web/main/.htaccess: use separate lowercase version when available: Generate the new dirname with a separate RewriteCond so its value can be used both in the -d test and in the replacement string, rather than separately for each
- 05:03 AM Revision 8056: web/main/.htaccess: translate dotpaths: Allow an unescaped . at the beginning of a filename, because this will never be a . separator. This adds support for hidden files in dir paths, which now won't be interpreted as dotpaths. However, regular files with extensions still need to have the filename escaped because it will otherwise be interpreted as a dotpath.
- 04:53 AM Revision 8055: web/main/.htaccess: Set Options +FollowSymLinks. It should be on by default ("All options except for MultiViews. This is the default setting." <http://httpd.apache.org/docs/2.2/mod/core.html#options>), but this makes sure it will always be enabled.
- 03:32 AM Revision 8054: web/main/.htaccess: Name the lowercased versions of dirs with a leading . (to make them hidden) instead of a trailing _ , to avoid having each dir listed twice in a row in the dir index
- 03:21 AM Revision 8053: Added web/main/TNRS/
- 03:15 AM Revision 8052: Added web/main/VegBank/
- 02:57 AM Revision 8051: Added web/main/BIEN2/
- 02:56 AM Revision 8050: web/main/index.php: Replaced - with . in namespaces to conform to new dotpath naming convention, which allows nesting of namespaces
- 02:35 AM Revision 8049: web/main/SALVIAS/.htaccess: Forward to new dd/ subdir
- 02:34 AM Revision 8048: Added web/main/SALVIAS/dd/
- 02:28 AM Revision 8047: web/main/DwC/.htaccess: Forward to new terms/ subdir
- 02:27 AM Revision 8046: Added web/main/DwC/terms/
- 01:51 AM Revision 8045: web/main/**/.htaccess: don't redirect subdir paths: Fixed bug where can only match non-empty string, because otherwise the rule would match this directory, which should still have its redirects processed
- 01:43 AM Revision 8044: web/main/index.php: Added back smaller spacing between the table columns
- 01:40 AM Revision 8043: web/main/main.css: blockquote: Removed right margin so there isn't a big space between the table columns in index.php, which results from nesting right-padded blockquotes inside one another
- 01:39 AM Revision 8042: web/main/index.php: Changed Brad-Boyle to just Brad because people's names only have to be unique within VegPath
- 01:32 AM Revision 8041: web/main/.htaccess: Added fallback redirect to VegCore for paths without a namespace. This can be used to link to specific VegCore terms without needing to include the VegCore namespace.
- 01:13 AM Revision 8040: Added web/main/VegBIEN/
- 01:13 AM Revision 8039: Added web/main/servers/vegbiendev/
- 01:13 AM Revision 8038: Added web/main/.phpPgAdmin/
- 01:12 AM Revision 8037: web/main/.phpMyAdmin/.htaccess: Set [redirect] flag in case the dest server is on the same machine as VegPath itself
- 12:25 AM Revision 8036: web/main/*/ lowercase versions: Renamed with _ suffix to avoid svn conflicts on case-insensitive filesystems such as Mac HFS+
- 12:24 AM Revision 8035: web/main/.htaccess: Support lowercase versions of mixed-case dirnames without breaking case-insensitive filesystems such as Mac HFS+
- 12:10 AM Revision 8034: web/main/: Added lowercase symlinks for mixed-case dirs to work with subdomain translation, which uses subdomains lowercased by DNS
- 12:08 AM Revision 8033: web/main/index.php: Use absolute URLs for dependencies to work with subdomain translation, which adds components to the URL path
03/15/2013
- 11:30 PM Revision 8032: web/main/**/.htaccess: Use RewriteRule instead of RedirectMatch to handle incremental redirects internally instead of issuing a (much slower) redirect to the web browser each time. This also handles edge cases better, as [last] RewriteRules can be used to control when to forward control to a subdir, and doesn't require prepending the path to the dir the .htaccess file is in. Note that this requires all gateway dirs (dirs with subdirs) to contain special RewriteRules to avoid redirecting subdir paths and handle DirectoryIndex; see web/main/DwC/.htaccess. Also note that the regexp of a catch-all RewriteRule must exactly follow the template for internal or external redirects; see web/main/SALVIAS/db/.htaccess for internal redirects and web/main/DwC/history/.htaccess for external redirects.
- 07:31 PM Revision 8031: web/main/.htaccess: translate dotpaths: Allow the part before the [] escape to contain [], to support labels that end in [] (like PHP array vars in the query string) labels with a simple array-subscript syntax (a[b]). This also shortens the regexp and makes it more readable without the \[\] in [^.\[\]/] . Note that this also allows invalid combinations of [] exprs (e.g. more than one per level or unbalanced []), which will still be translated but will probably not have the desired result.
- 07:16 PM Revision 8030: web/main/.htaccess: translate dotpaths: Inline the [] escape regexp into the main regexp, because it is now approximately the same length as the []-matching portion of the main regexp and this greatly simplifies the code by removing the extra RewriteCond. Note that the translation rule is now a plain regexp (run repeatedly until no match), which can be used in *any* programming language that supports Perl-compatible regular expressions, not just mod_rewrite.
- 07:06 PM Revision 8029: web/main/.htaccess: translate dotpaths: discardpath explanation: Clarified that the infinite loop resulted from reappending PATH_INFO (the Apache-matched filename)
- 07:02 PM Revision 8028: web/main/.htaccess: translate dotpaths: Require any [] escape to have the ] at the end of the level, to simplify the [] regexp
- 06:54 PM Revision 8027: web/main/.htaccess: translate dotpaths: Use a lookahead assertion to ensure that at least one character is matched as the head of the dotpath. This ensures that (.*/)? + the rest of the regexp does not match a path with a trailing /, which is a sealed /-path and not subject to dotpath translation.
- 06:46 PM Revision 8026: web/main/.htaccess: translate dotpaths: Only support one [] escape per dot-level to (greatly) shorten the [] regexp. This does not pose a problem for encoding . because the entire level can simply be enclosed in [].
- 06:17 PM Revision 8025: web/main/.htaccess: Uncommented ErrorDocument
- 06:15 PM Revision 8024: web/main/.htaccess: translate dotpaths: Removed separate sealing of the /-path, which is now performed by the main RewriteRule because it appends a / even if there is no . suffix. This does not cause an infinite loop because a character is always added (/), which prevents the previously-matched head (after the last / but before any .) from being matched again.
- 06:05 PM Revision 8023: web/main/.htaccess: translate dotpaths: Fixed bug where it's actually the portion *before* the . (but after the last /) that should be subject to []-unescaping, rather than the portion *after* the . . Fixed bug where [] escapes were not being unescaped because the wildcard .* group matched the whole head portion instead of allowing the []-captures to match.
- 03:17 PM Revision 8022: web/main/.htaccess: translate subdomain to path: Don't use expr RewriteConds because they are not supported by Apache 2.2. Instead issue an external redirect with the subdomain part of the hostname removed, for the purpose of changing HTTP_HOST so that the replacement is not performed again if the mod_rewrite rules are run more than once.
- 02:46 PM Revision 8021: web/main/**/.htaccess: Use RewriteEngine, inheriting the web/main/.htaccess rules, in order to translate dotpaths that follow /-paths to existing dirs
- 02:45 PM Revision 8020: web/main/DwC/.htaccess: Moved DwC.history redirect to web/main/DwC/history/.htaccess
- 02:40 PM Revision 8019: web/main/SALVIAS/db/.htaccess: Removed trailing / because for DB redirects, this is apparently necessary
- 02:30 PM Revision 8018: web/main/.phpMyAdmin/.htaccess: Prepend http:// to the dest URL stem, instead of requiring the dest URL to provide the protocol, because the two // are replaced with one / by Apache when mod_rewrite is on, creating an invalid URL
- 01:55 PM Revision 8017: web/main/.htaccess: translate dotpaths: Fixed bug where can't remove all [] escapes, because this would also remove [] from protected parts of the URL (i.e. before the last /). Instead just remove up to two [] escapes per dot-level, e.g. a.[b.c]=[d.e] -> a/b.c=d.e . Note that this removes the restriction against URLs containing "[]", because these [] will be left as-is as long as they are followed at some point by a / .
- 01:41 PM Revision 8016: Removed web/.htaccess because it is not part of any site (web/main/.htaccess is the top-level .htaccess file for the default site)
- 01:17 PM Revision 8015: web/main/.htaccess: translate dotpaths: replace all unescaped . with / : Allow empty components on either side of the . , to ensure that *all* unescaped dots are consistently replaced with / . Note that hidden directories that start with . will have the . preserved, because they have (or get) a trailing slash, which prevents dotpath from operating on them.
- 01:13 PM Revision 8014: web/main/.htaccess: translate dotpaths: remove all [] escapes: Also remove empty [] to allow permalinking clients to easily generate escaped strings by URL-encoding the value and enclosing it in [], which will now also work even if the value is empty. This change will break any destination URLs that use "[]", such as PHP scripts that use this to indicate an array variable in the query string. In these URLs, the [] must now be %-encoded.
- 01:00 PM Revision 8013: web/main/.htaccess: RewriteRules with [redirect] and http:// : Removed [redirect] because it is implied when the replacement is an absolute URL
- 12:57 PM Revision 8012: web/main/.htaccess: RewriteRules with [redirect]: Added last because apparently redirect will not immediately cause the redirect, and will instead continue rewriting the URL ("You will almost always want to use [R] in conjunction with [L] (that is, use [R,L]) because on its own, the [R] flag prepends http://thishost[:thisport] to the URI, but then passes this on to the next rule in the ruleset, which can often result in 'Invalid URI in request' warnings." <http://httpd.apache.org/docs/2.2/rewrite/flags.html#flag_r>)
- 12:34 PM Revision 8011: web/main/.htaccess: RewriteRules: Added [discardpath,noescape] flags to all rules, because these are needed for correct interpretation of the replacement and should be the default setting. For some rules they may be optional, but it is better to include them on all rules to ensure there is never an unexpected effect from omitting them.
- 12:29 PM Revision 8010: web/main/.htaccess: Added "RewriteOptions inherit" because this[1] should be the default setting, to ensure that global filter rules also occur in subdirs. Note that the presence of this directive in *this* .htaccess file does not solve this problem, but putting it there will remind writers of subdir .htaccess files that they need to include that directive.
- [1] The value should really be InheritBefore, but this is not supported in Apache 2.2, which we have (http://httpd.ap...
- 12:02 PM Revision 8009: web/main/SALVIAS/.htaccess: RedirectMatch for below top-level: Use /(.*?)/?$ suffix for consistency with other sources' RedirectMatch patterns
- 12:00 PM Revision 8008: web/main/TCS,VegX/.htaccess: Fixed bug where need to match dirs by themselves in addition to when followed by a filename
- 11:58 AM Revision 8007: web/main/DwC/.htaccess: Fixed bug where need to match dirs by themselves (/DwC/, /DwC/history/) in addition to when followed by a filename
- 11:44 AM Revision 8006: web/main/.htaccess: mod_rewrite: Removed unneeded RewriteBase, because the default setting for RewriteBase is actually the dir of the .htaccess file itself, rather than always / . ("RewriteRule can be used in per-directory config files (.htaccess). There it will act locally, i.e., the local directory prefix is stripped at this stage of processing and your rewriting rules act only on the remainder. At the end it is automatically added back to the path." <http://httpd.apache.org/docs/1.3/mod/mod_rewrite.html#RewriteBase>)
- 11:30 AM Revision 8005: web/main/SALVIAS/*/.htaccess: Added trailing / to Redirect paths, which works now that a trailing / is automatically added to all paths by dotpath translation
- 11:28 AM Revision 8004: web/main/SALVIAS/: Moved db/ and users/ actions to their corresponding subdirs. The rules will still be run in the appropriate order, because if a .htaccess exists in a subdir, it will be used in preference to the .htaccess in the parent dir.
- 10:26 AM Revision 8003: my2pg: Removed SETs for standard_conforming_strings, escape_string_warning because these are also prepended by my2pg.data, which this script's output is piped to
- 10:00 AM Revision 8002: my2pg.data: \0 removal: Fixed bug where need to handle all backslashes before the 0, because they might escape just each other (an even number of \) rather than also encoding the \0 (an odd number of \)
- 08:20 AM Revision 8001: web/main/.htaccess: translate dotpaths to /-paths: Added examples
- 08:04 AM Revision 8000: web/main/.htaccess: translate dotpaths to /-paths: Translate *all* .-separated components in the path to / instead of just those that exist in the filesystem. This allows per-source rules to match just a / for the path-element separator instead of sometimes / and other times . (depending on whether that portion of the dotpath had been translated). Support [] escapes that preserve any . they surround, e.g. a.[b.c] -> a/b.c . (Supporting []-escapes requires that the /-path be "sealed" by appending a / , to prevent the . -> / translation rule from reinterpreting a newly-unescaped [] sequence as a dotpath. This also requires that the translation rule ignore anything before the last /, because it could have been []-unescaped by a previous round of mod_rewrite, e.g. in another dir. All the per-source rules need to be retrofitted to support the new trailing / .) Note also the discardpath flag (http://httpd.apache.org/docs/2.2/rewrite/flags.html#flag_dpi) on all RewriteRules that perform a replacement on the input string (i.e. that have a pattern other than ^.*$). This flag is needed to avoid infinite loops, because otherwise, a critical bug in mod_rewrite causes it to *reappend* the filename portion of the input string (PATH_INFO) to the result, causing it to be present twice, e.g. a/b -> a/b/b (https://issues.apache.org/bugzilla/show_bug.cgi?id=38642).
- 03:09 AM Revision 7999: web/main/nimoy/: Moved to web/main/servers/ so it can be grouped with other resources that are servers
03/14/2013
- 11:57 PM Revision 7998: web/main/index.php: $root: Fixed bug where need to use gethostbyaddr($_SERVER["SERVER_ADDR"]) instead of $_SERVER["SERVER_NAME"] because SERVER_NAME is actually set to whichever hostname the server is accessed through (e.g. path.vg or vegbiendev.nceas.ucsb.edu), rather than always being set to the reverse-DNS lookup of SERVER_ADDR, as is specified in the Apache docs ("If no ServerName is specified, then the server attempts to deduce the hostname by performing a reverse lookup on the IP address" <http://httpd.apache.org/docs/2.2/mod/core.html#servername>)
- 11:01 PM Revision 7997: web/main/.htaccess: Added back "don't rewrite existing paths" to ensure that rules after it will not rewrite any actual files (web page dependencies)
- 10:50 PM Revision 7996: web/main/nimoy/.htaccess: Removed no longer needed trailing /
- 10:49 PM Revision 7995: web/main/.phpMyAdmin/.htaccess: Add trailing / to provided URL in case the user didn't add it
- 10:46 PM Revision 7994: web/main/nimoy/.htaccess: Removed unnecessary index.php from end of phpMyAdmin URL
03/13/2013
- 11:09 PM Revision 7993: web/main/DwC,SALVIAS,TCS,VegX: Added .htaccess redirects and dot-path-matching empty dirs, which will replace the index.php redirects with a simpler syntax (http://httpd.apache.org/docs/2.2/mod/mod_alias.html#redirect)
- 09:15 PM Revision 7992: web/main/.htaccess: translate dot-paths to dirs: Documented that this only occurs when the dirs exist in the filesystem, and that each subdir Redirect directive needs a corresponding tree of (empty) dirs
- 08:49 PM Revision 7991: Removed no longer used web/main/_archive/index.htm. Use web/main/index.php instead, which issues the Location: header to support clients other than web browsers.
- 08:37 PM Revision 7990: web/vegpath/: Renamed to web/main/ because the content here is the main site hosted on vegbiendev. It will eventually contain internal links to VegBIEN resources in addition to external links (persistent URLs) to sources used in creating VegBIEN.
- 07:45 PM Revision 7989: web/vegpath/.htaccess: Removed "don't rewrite existing paths" RewriteRule because there are no longer any rules after it that require it
- 07:31 PM Revision 7988: web/vegpath/.htaccess: translate dot-paths to dirs: translate all levels of dot-paths present in the filesystem rather than just the first level, by using the [next] RewriteRule flag (http://httpd.apache.org/docs/current/rewrite/flags.html#flag_n). This avoids the need to translate further levels of dot-paths in each subdir's .htaccess.
- 07:07 PM Revision 7987: web/vegpath/.htaccess: translate subdomain to path: Simplified using new subdomain2path RewriteMap, which avoids the need for a separate RewriteRule+RewriteConds for each number of nested subdomains
- 07:06 PM Revision 7986: web/vegpath.conf: subdomain2path RewriteMap: Fixed bug where need `RewriteEngine on` in the VirtualHost config for a prg: RewriteMap to take effect ("External rewriting programs are not started if they're defined in a context that does not have RewriteEngine set to on." <http://httpd.apache.org/docs/current/rewrite/rewritemap.html#prg>)
- 06:00 PM Revision 7985: web/vegpath.conf: Added subdomain2path RewriteMap
- 06:00 PM Revision 7984: Added web/subdomain2path for use by RewriteMap
- 05:49 PM Revision 7983: web/vegpath.conf: Added spacing between sections and reordered directives
- 05:43 PM Revision 7982: web/vegpath/.htaccess: translate 2-level subdomain to prefix: interpret subdomains in (standard) reverse instead of forward order (e.g. history.DwC instead of DwC.history)
- 05:39 PM Revision 7981: web/vegpath/.htaccess: Don't also translate dash-paths to dirs (in addition to dot-paths) because this adds unnecessary complexity and was only used to allow expressing parts of subdomain paths in forward instead of reverse order (e.g. DwC-history instead of history.DwC)
- 05:31 PM Revision 7980: web/vegpath/.htaccess: translate dot-/dash-paths to dirs: don't match / inside a . or - component, because it has a lower precedence
- 05:31 PM Revision 7979: web/vegpath/.htaccess: translate dot-/dash-paths to dirs: changed comment to indicate that these rules can (eventually) translate any dot-/dash-path, not just the head
- 04:30 PM Revision 7978: web/vegpath/.htaccess: translate subdomain to prefix: Use last to avoid further rewriting, since these rules make the destination a /-path
- 04:28 PM Revision 7977: web/vegpath/.htaccess: Also translate dash-paths (e.g. DwC-history) to dirs
- 10:41 AM Revision 7976: web/vegpath/.htaccess: translate both 1-level and 2-level subdomains to dir prefixes
- 10:39 AM Revision 7975: web/vegpath/.htaccess: translate subdomain to prefix: HTTP_HOST RewriteCond: Don't capture part after . because it isn't used as a replacement (it used to be used for this when a now-unnecessary browser redirect was performed)
- 10:33 AM Revision 7974: web/vegpath/.htaccess: don't rewrite existing paths: Moved before other RewriteRules so that page dependencies (such as util.js) can be loaded more quickly without needing to pass through all the rewrite rules first
- 10:04 AM Revision 7973: web/vegpath/.htaccess: ErrorDocument: Use / instead of /index.php because index.php is the DirectoryIndex and doesn't need to be hardcoded
- 09:44 AM Revision 7972: web/vegpath/.htaccess: translate subdomain to prefix: Fixed bug where can only run this is if the dest subdir has a .htaccess file, because this prevents the RewriteRule from being re-run for the subdir (which would lead to infinite recursion because the rule is not idempotent)
- 09:37 AM Revision 7971: web/vegpath/.htaccess: translate subdomain to prefix: Fixed bug where need to prepend %{DOCUMENT_ROOT} to the file path when testing it for existence
- 09:16 AM Revision 7970: web/vegpath/.htaccess: translate path head to dir (if the dir exists)
- 08:56 AM Revision 7969: web/vegpath/.htaccess: translate subdomain to prefix: Set nosubreq to avoid infinite recursion (the RewriteRule is not idempotent)
- 08:43 AM Revision 7968: web/vegpath/.htaccess: translate subdomain to prefix: Don't translate all nested subdomains, because some of them may result from the machine's own name (vegbiendev.nceas.ucsb.edu) and should not be removed. Instead, remove only the leftmost level of subdomain, and only if it corresponds to a filesystem directory.
- 08:39 AM Revision 7967: web/vegpath/index.php: Use $alias whenever VegPath was reached via the machine's own name (vegbiendev.nceas.ucsb.edu) rather than via a shortened domain (path.vg), so that the right path root can seamlessly be used when path.vg is down
- 07:50 AM Revision 7966: web/vegpath/.htaccess: Translate subdomain to path prefix
- 07:30 AM Revision 7965: Added inputs/MO/_src/TropicosSpecimens_2013_03_12_*.txt.md5
- 07:25 AM Revision 7964: Added inputs/MO/_archive
- 07:25 AM Revision 7963: Added inputs/MO/_src/
- 07:14 AM Revision 7962: web/vegpath/index.php: Brad-Boyle root: Fixed link
- 07:12 AM Revision 7961: web/vegpath/index.php: CTFS root: Fixed link
- 07:11 AM Revision 7960: web/vegpath/index.php: CTFS-schema: Use $Redmine_svn
- 06:46 AM Revision 7959: web/vegpath/.htaccess: Remove www subdomain
- 06:25 AM Revision 7958: Added web/logs/
- 06:22 AM Revision 7957: web/vegpath/.htaccess: Use the simpler ErrorDocument directive instead of a catch-all RewriteRule to handle non-filesystem paths
- 06:11 AM Revision 7956: web/vegpath/index.php: $alias: Use HTTP_HOST and SCRIPT_NAME so the alias doesn't need to be hardcoded
- 06:09 AM Revision 7955: web/vegpath/util.php: Added ensure_prefix(), ensure_suffix()
- 05:32 AM Revision 7954: web/vegpath/index.php: Switching back to path.vg now that the .vg root DNS is back online
- 04:36 AM Revision 7953: web/vegpath/index.php: Use REQUEST_URI (removing the script's own dir) instead of PATH_INFO, to support being used for the Apache ErrorDocument directive
- 04:24 AM Revision 7952: web/vegpath/.htaccess: mod_rewrite: Don't run any further rewrite rules if path exists, to avoid needing to add RewriteConds to *each* RewriteRule to check if the path exists
- 01:39 AM Revision 7951: web/vegpath/index.php: Changed page title to contain "persistent URLs" because VegPaths are conceptually a type of PURL <http://en.wikipedia.org/wiki/PURL>
- 12:04 AM Revision 7950: web/vegpath/index.php: $alias: Switching back to j.mp/vegpath# because the .vg root DNS server is down. (The .vg registrar adamsnames.com also displays an under construction page.) If the registrar stays down, we may want to switch a different top-level domain.
03/12/2013
- 07:59 PM Revision 7949: web/vegpath/.htaccess: DirectoryIndex: Removed index.htm because HTML files should not be used for redirection (they don't issue the Location: header needed by clients other than web browsers)
- 07:52 PM Revision 7948: web/vegpath/.htaccess: mod_rewrite: Added `RewriteOptions inherit` so this .htaccess file can be copied to subdirs and will inherit the "unknown paths" rule
- 07:37 PM Revision 7947: web/vegpath/index.php: Use $PATH_INFO (i.e. index.php/$PATH_INFO) instead of the query string to pass the path to index.php, so that the query string can be used for other things
- 04:28 AM Revision 7946: web/vegpath/index.php: top-level path parsing: Use parse_dot_path() instead of parse_mixed_path() because XPaths are now (client-side) fragments, and don't need to be supported by the path mechanism
- 04:26 AM Revision 7945: web/vegpath/index.php: SALVIAS: Don't append #Plot_ when there is no subpath
- 03:19 AM Revision 7944: web/vegpath/index.php: SALVIAS: Factored out URL to $SALVIAS
- 03:16 AM Revision 7943: web/vegpath/index.php: SALVIAS: Appended Plot_ to dest URL because both data dictionary sections start with this, so it doesn't need to be included in the VegPath
- 02:56 AM Revision 7942: web/vegpath/index.php: XPath resources: Removed # at end of destination URL because the # is now part of the #/XPath suffix (and is added by the user)
- 02:53 AM Revision 7941: web/vegpath/index.php: XPath resources: Added # before XPath because the XPath is actually a location *within* the resource, and thus resolvable by the client (or user) themselves, rather than part of the redirected URL (which only needs to forward to the XML schema file)
- 02:41 AM Revision 7940: web/vegpath/index.php: Updated $alias to the newly-registered path.vg domain, which is shorter and more intuitive-looking than the previous value j.mp/vegpath (from bitly.com). Note that using a domain also allows redirects without JavaScript, which can be used by any web client (such as a Python script), not just a web browser.
03/11/2013
- 11:19 PM Revision 7939: web/vegpath/index.php: Renamed ref to ns to emphasize that the resources in this field are namespaces for the paths
- 11:16 PM Revision 7938: web/vegpath/index.php: add_ref(): Renamed $suffix to $path_pat to distinguish it from the actual suffix string sent in the URL
- 11:09 PM Revision 7937: web/vegpath/index.php: custom_separator(): Marked optional params as such
- 11:07 PM Revision 7936: web/vegpath/index.php: Terms: IH-db: Factored out phpMyAdmin URL into $IH_db
- 11:05 PM Revision 7935: web/vegpath/index.php: Terms: Added IH-db
- 11:05 PM Revision 7934: web/vegpath/index.php: php*Admin(): Added support for a fixed table that doesn't need to be specified in the path
- 10:43 PM Revision 7933: web/: Removed svn:executable permissions from non-executable files
- 10:36 PM Revision 7932: web/vegpath/index.php: Data: Added Institutions subsection
- 10:33 PM Revision 7931: web/vegpath/index.php: Placed Terms and Data sections side-by-side in two columns so that data resources would not need to come after a long list of terms resources
- 10:31 PM Revision 7930: web/vegpath/main.css: Added .indent
- 10:11 PM Revision 7929: web/vegpath/index.php: People: Added Brad's VegCore ID terms spreadsheets
- 10:10 PM Revision 7928: web/vegpath/index.php: Factored Redmine svn URLs out into $Redmine_svn
- 10:06 PM Revision 7927: Added schemas/VegCore/Brad_Boyle/ with Brad's VegCore IDs terms spreadsheets
- 09:51 PM Revision 7926: web/vegpath/index.php: URL fragment redirect: Removed loading indicator (blinking of resource being loaded) because for file downloads, the download will not redirect to a new page and the resource name will instead remain blinking
- 09:48 PM Revision 7925: web/vegpath/index.php: URL fragment redirect: Fixed bug where need to use parse_mixed_path() instead of parse_dot_path() to support XPaths
- 09:48 PM Revision 7924: web/vegpath/index.php: Terms: Added CTFS
- 09:23 PM Revision 7923: web/vegpath/index.php: URL patterns header: Reworded comment to clarify that the elements that can be left out are in italics
- 09:09 PM Revision 7922: web/vegpath/index.php: URL fragment redirect: Only perform fragment redirect if URL shortener requires it (as indicated by the $alias ending in #)
- 09:01 PM Revision 7921: web/vegpath/index.php: nimoy nodes: Factored nimoy phpMyAdmin() calls out into nimoy_db() function
- 08:57 PM Revision 7920: web/vegpath/index.php: nimoy nodes: Factored nimoy URL prefix out into $nimoy
- 08:51 PM Revision 7919: web/vegpath/index.php: SALVIAS: Added subnodes for each SALVIAS database on nimoy
- 08:37 PM Revision 7918: web/vegpath/index.php: VegCore: Changed suffix from .table.column to .term because VegCore terms are globally unique and don't need a table prefix
- 08:36 PM Revision 7917: web/vegpath/index.php: BIEN2: Added subnodes for each BIEN2 database on nimoy
- 08:04 PM Revision 7916: web/vegpath/index.php: Inlined VegBank() because it's only used once
- 08:01 PM Revision 7915: web/vegpath/index.php: VegBank(): Use custom_separator()'s new $main_url param
- 08:00 PM Revision 7914: web/vegpath/index.php: custom_separator(): Added $main_url param like in by_prefix(), for use when there is no path
- 07:55 PM Revision 7913: web/vegpath/index.php: VegBank(): Use new custom_separator()
- 07:55 PM Revision 7912: web/vegpath/index.php: VegBank(): Use new custom_separator()
- 07:53 PM Revision 7911: web/vegpath/index.php: VegBank(): Use new custom_separator()
- 07:53 PM Revision 7910: web/vegpath/index.php: Added custom_separator()
- 07:52 PM Revision 7909: web/vegpath/index.php: Add # suffix to $alias instead of adding it in add_ref(), because not all aliases will use a # suffix
- 07:48 PM Revision 7908: web/vegpath/util.php: Added array_non_empty(), join_non_empty()
- 07:14 PM Revision 7907: web/vegpath/index.php: add_ref(): redirect: Don't display "Loading" as the page itself is not shown to the user when using a Location: header
- 07:12 PM Revision 7906: web/vegpath/index.php: URL fragment redirect: Fixed bug where can't prefix the fragment with / because this won't work if VegPath is not at the top-level of the domain (and in any case is unnecessary to form a relative path)
- 07:09 PM Revision 7905: web/vegpath/index.php: URL fragment redirect: Blink the namespace being loaded, rather than displaying "Loading" at the bottom of the page, to make it clearer what's being loaded
- 06:42 PM Revision 7904: web/vegpath/index.php: add_ref(): Don't display the URL pattern in monospace, because fixed-width font is not needed and variable-width font is easier to read. This requires decreasing the header level because variable-width font is bigger than fixed-width font.
- 06:41 PM Revision 7903: web/vegpath/main.css: Added entries for h6
- 06:28 PM Revision 7902: web/vegpath/index.php: parse_dot_path() calls: Fixed bug where need to test result components using \`!== ""\` instead of just using the value as a boolean because PHP incorrectly interprets the string "0" as false
- 06:24 PM Revision 7901: web/vegpath/util.php: Path: Made $tail default to ""
- 06:20 PM Revision 7900: web/vegpath/index.php: URL fragment redirect: Prepend / instead of ? because filesystem paths are now redirected using .htaccess, removing the need to set the query string
- 06:16 PM Revision 7899: web/vegpath/index.php: phpMyAdmin(): Fixed bug where can't use tbl_alter as the column target page because it requires a token param, which is only obtained after the user logs in
- 06:04 PM Revision 7898: web/vegpath/util.php: partition(): Fixed bug where strpos() actually does take the search-for param second rather than first (but returns false on no match)
- 06:00 PM Revision 7897: web/vegpath/util.php: partition(): Fixed bug where strpos() returns false rather than -1 on no match
- 05:59 PM Revision 7896: web/vegpath/util.php: partition(): Fixed bug where strpos() takes the search-for param first rather than second
- 05:37 PM Revision 7895: web/vegpath/index.php: php*Admin(): Append $target to $url in separate step from returning $url, to make it easier to debug-print the value of $url before returning it
- 05:30 PM Revision 7894: web/vegpath/index.php: BIEN2: Fixed to use phpMyAdmin with correct URL instead of vegbiendev phpPgAdmin placeholder
- 05:20 PM Revision 7893: web/vegpath/index.php: Added phpMyAdmin()
- 04:59 PM Revision 7892: web/vegpath/.htaccess: mod_rewrite: RewriteBase: Changed to / now that VegPath is hosted at vegbiendev.nceas.ucsb.edu
- 04:58 PM Revision 7891: web/vegpath/.htaccess: unknown paths: Don't redirect to index.php if the path already starts with index.php, to help avoid infinite redirect recursion
- 04:54 PM Revision 7890: web/vegpath/.htaccess: unknown paths: Removed no longer accurate VegRef from comment header
- 04:43 PM Revision 7889: vegpath/.htaccess: DirectorySlash: Lowercased On for consistency with other boolean directives
- 04:26 PM Revision 7888: Added web/vegpath.conf with Apache VirtualHost config
- 03:26 PM Revision 7887: vegpath/index.php: VegBank(): Fixed bug where name had incorrectly been replaced with $name
- 03:24 PM Revision 7886: vegpath/index.php: phpPgAdmin(): Factored out appending of "&subject="
- 03:23 PM Revision 7885: vegpath/: Archived no longer used vegpath/index.htm
- 03:05 PM Revision 7884: vegpath/index.*: IH: URL-encode ' in search URL to avoid needing to backslash-escape it
- 03:04 PM Revision 7883: vegpath/index.php: IH: Fixed search URL to undo incorrect search-and-replace of + with .
03/10/2013
- 06:51 AM Revision 7882: vegpath/index.*: Changed "Supported URLs" to "Supported URL patterns" because the provided short URLs take parameters that are filled into the destination URL
- 06:46 AM Revision 7881: vegpath/index.*: Removed tagline from window title because the window title (which becomes the bookmark name) only needs to be long enough to identify what the page is, and should not be cluttered with descriptions
- 06:43 AM Revision 7880: vegpath/index.*: Changed tagline to emphasize that the shortened URLs are especially intended to be human-readable, e.g. for use as column names
- 06:37 AM Revision 7879: VegRef/: Renamed to VegPath to make it clearer that the purpose of the web service is to provide short, globally unique paths to resources (e.g. for use as global IDs <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Global_IDs>)
- 06:10 AM Revision 7878: VegRef/.htaccess: URL resolution: mod_rewrite: Fixed bug where also need to consider if %{REQUEST_FILENAME} is a dir or symlink, which occurs e.g. when redirecting to the VegRef/ dir itself
03/09/2013
- 10:50 PM Revision 7877: web/VegRef/.htaccess: Turned off RewriteEngine temporarily because it was preventing the internal VegRef redirection from working
- 10:48 PM Revision 7876: Added web/VegRef/, which provides a URL-shortening service for vegetation references. VegRef URLs are short and human-readable enough to be used as database column names. This also causes Google spreadsheets to display a clickable link for each column name, which takes the user to the data dictionary definition for the term. String row IDs (such as global IDs <https://projects.nceas.ucsb.edu/nceas/projects/bien/wiki/Global_IDs>) can also be clickable VegRef URLs.
03/08/2013
- 10:26 PM Revision 7875: inputs/NCU/Specimen/map.csv: cultivated: Map false (0) to NULL because cultivated can only be true or NULL, not false
- 09:53 PM Task #584 (Resolved): enable running all the import steps from one runscript
- see [[Import process]]
- 09:14 PM Revision 7874: inputs/CTFS/_src/_README.TXT: Added e-mails from Brad Boyle about issues with species aggregation methods
03/07/2013
- 06:27 PM Revision 7873: inputs/UNCC/Specimen/map.csv: Populated specimenHolderInstitutions
- 03:21 PM Revision 7872: Added planning/goals/BIEN3_derived_data_products.docx.url
- 03:19 PM Revision 7871: Added planning/goals/BIEN3_derived_data_products.docx
03/06/2013
- 11:00 PM Revision 7870: inputs/ARIZ/omoccurrences/map.csv: cultivationStatus: Map to NULL if it's false, because cultivated can only be true or NULL
- 10:57 PM Task #581 (New): mechanism to dynamically autopopulate NOT NULL values
- * this is currently done using sql_io.put_table()'s col_defaults parameter, which only supports constant values
* th... - 10:01 PM Revision 7869: schemas/vegbien.sql: analytical_stem_view: Merged taxonName_verbatim and scientificNameAuthorship_verbatim into scientificName_verbatim
- 09:29 PM Revision 7868: schemas/vegbien.sql: analytical_plot: Added specimens fields at the end, for use by plots data that also tracks specimens (e.g. Madidi, SALVIAS)
- 09:21 PM Revision 7867: README.TXT: Full database import: In PostgreSQL: Removed tests related to unscrubbed_taxondetermination_view and TNRS taxondeterminations, which are no longer needed because TNRS is joined directly to analytical_stem_view
- 09:08 PM Task #580 (New): automap each word *in* a VegCore term instead of just the whole term
- * this avoids the need to create repetitive synonyms for each variation of a term (e.g. with different units)
- 08:45 PM Task #579 (New): in sql_gen.EnsureNotNull, use ARRAY[] instead of COALESCE() to wrap values where NULL should be compared literally
- * this avoids the need for different null sentinels for each datatype
* it would still be necessary to ensure that t... - 08:36 PM Revision 7866: mappings/VegCore.htm: Regenerated from wiki. Renamed specimenDonor to specimenDonorInstitution to clarify that this is an institution, not a collector.
- 08:30 PM Revision 7865: inputs/UNCC/Specimen/map.csv: Mapped flower, fruit -> reproductiveCondition using translation is from e-mail conv between Jim Matthews at Habitat Assessment and Bob Peet on 2010-12-2~3
- 08:17 PM Revision 7864: make_analytical_db: Support the $log flag, which toggles output to the terminal or a log file
- 07:50 PM Revision 7863: inputs/import.stats.xls: Updated import times. The core import time has dropped by more than half (!) to ~12 hours, now that the TNRS scrubbing is added using a simple LEFT JOIN, instead of being pushed through the normalized schema. Not since October has the import been this fast!
- 07:27 PM Revision 7862: import_times: When there are no matching logs, exit with status 0 instead of 1 because this is a normal condition
- 07:26 PM Revision 7861: import_times: Fixed bug where sed would wait for user input from stdin when there were no matching logs
- 07:22 PM Revision 7860: README.TXT: Full database import: record the import times in inputs/import.stats.xls: If the row counts do not match up: Added step to get the datasource names from the log file footers
- 07:12 PM Revision 7859: inputs/UNCC/Specimen/map.csv: cultivated: assume anything non-NULL to be cultivated, so that we err on the side of excluding possibly-cultivated specimens
- 06:49 PM Revision 7858: inputs/UNCC/_src/_README.TXT: Added e-mail conv between Jim Matthews at Habitat Assessment and Bob Peet about the meanings of the Leaf, Flower, Fruit, and Root codes
- 05:38 PM Revision 7857: README.TXT: Full database import: Added steps to manually back up the DB. (These steps are specific to the import process, and include filters to exclude previous imports. There are also general backup steps in the Backups > Full DB section.)
- 02:56 PM Revision 7856: inputs/ACAD/Specimen/map.csv: institutionCode: Replace "Acadia University" with the IH code ACAD
- 02:18 PM Revision 7855: inputs/.TNRS/data.sql: Re-ran TNRS on the test_taxonomic_names so that the sample data would contain all the accepted names under the current acceptedScientificName formula, and would not produce any entries for unscrubbed names in tnrs_input_name
- 02:08 PM Task #578 (New): replace all fields of type text with a domain type that ensures the value is non-empty (not '')
- * empty strings tend to cause bugs all over the database because they are sometimes, but not always, treated as NULL
- 01:59 PM Revision 7854: schemas/vegbien.sql: tnrs_input_name: taxonlabel.taxonomicname subquery: Don't exclude taxonlabels with a matched_label_id, because matched_label_id is no longer used to indicate that a name has a scrubbed name (this was used when all of TNRS was imported at the beginning of the import)
- 01:56 PM Revision 7853: schemas/vegbien.sql: tnrs_input_name: accepted_name subquery: Renamed to the VegCore table name AcceptedTaxon
- 01:54 PM Revision 7852: schemas/vegbien.sql: tnrs_input_name: taxonlabel.taxonomicname subquery: Use new MatchedTaxon view, which use VegCore column names where possible
- 01:46 PM Revision 7851: schemas/vegbien.sql: tnrs_input_name: accepted_name subquery: Use new MatchedTaxon, AcceptedTaxon views, which use VegCore column names where possible
- 01:22 PM Revision 7850: README.TXT: Full database import: Added instructions to run TNRS and view the TNRS log
- 10:21 AM Revision 7849: README.TXT: Full database import: Removed no longer necessary command to scrub unscrubbed taxondeterminations
Also available in: Atom