Added inputs/FIA/COUNTY/postprocess.sql

Added inputs/FIA/REF_PLANT_DICTIONARY/postprocess.sql

inputs/FIA/COND/postprocess.sql: Matched COND.HABTYPCD1, COND.HABTYPCD1_PUB_CD to REF_HABTYP_DESCRIPTION

inputs/input.Makefile: Staging tables installation: $(exportHeader): Fixed bug where need to run postprocess.sql before exporting the header, because it can change the column names

inputs/input.Makefile: Staging tables installation: $(exportHeader): export the header before running $(cleanup), because the header is not affected by the data cleanup operations and thus can be generated right away, to allow mapping while the cleanup operations run

inputs/FIA/REF_HABTYP_DESCRIPTION/postprocess.sql: Prepare columns for joining with COND

inputs/input.Makefile: Staging tables installation: $(exportHeader): Fixed bug where need to use psql_script_vegbien instead of the psql_verbose_vegbien used by $(psqlAsBien), to avoid echoing commands as part of the exported header

Added planning/workflow/(de)normalized_import.mappings.png

Added planning/workflow/denormalized_import.png, normalized_import.png

web/main/IH/: Added lowercase alias

Added web/main/IH/

inputs/input.Makefile: Staging tables installation: Added postprocess target, which runs all the postprocess.sql files

inputs/FIA/REF_SPECIES/postprocess.sql: Cast ID column to integer

inputs/FIA/*/postprocess.sql: Cluster tables by their *.unique index for faster joins

inputs/FIA/*/postprocess.sql: Cast ID columns to integer using new functions.set_col_types()

bin/psql_verbose_vegbien: Run with client_min_messages = NOTICE to display notices for debugging. This is supposed to be the default, but apparently isn't.

inputs/input.Makefile: BIEN commands: $(psqlAsBien): Use psql_verbose_vegbien instead of psql_script_vegbien so that timings and notices are displayed, which is useful for profiling and debugging

schemas/functions.sql: Added col_cast and set_col_types()

schemas/functions.sql: Added col_ref, col_type()

schemas/functions.sql: Added cluster_once()

schemas/functions.sql: Added cluster_index()

schemas/functions.sql: create_if_not_exists(): Also handle duplicate_column exceptions

schemas/functions.sql: Added rename_if_exists()

inputs/FIA/COND/postprocess.sql: Renamed oldgrowth to COND.oldgrowth so it wouldn't be renamed by to_global_col_names()

inputs/FIA/COND/postprocess.sql: Added oldgrowth column as part of the postprocessing instead of as part of the view that left joins the core tables together. This avoids needing to regenerate the oldgrowth field whenever the view is queried or materialized.

inputs/FIA/TREE/postprocess.sql: Added index on columns that join to parent tables

inputs/FIA/*/postprocess.sql: Removed table prefix from globally-unique columns that should be joined on

schemas/functions.sql: Marked STRICT functions as such

schemas/functions.sql: col_global_names(): Treat any column name that contains . as already being globally unique, and don't prepend the table name. This allows renaming the table columns after running col_global_names(), without causing the table name to be re-prepended the next time col_global_names() is run.

schemas/functions.sql: Added contains()

schemas/functions.sql: Added create_if_not_exists()

inputs/FIA/*/postprocess.sql: Use functions.to_global_col_names() to ensure that all column names are globally unique. This makes it easy to join the tables together without worrying about column name collisions.

inputs/FIA/*/postprocess.sql: Use new functions.create_if_not_exists() to allow re-running postprocess.sql idempotently

inputs/input.Makefile: Staging tables installation: %/install: Use new %.sql/run to run postprocess.sql

inputs/input.Makefile: Staging tables installation: Added %.sql/run to run postprocess.sql, etc. separately from the install targets they are a part of

inputs/input.Makefile: Staging tables installation: Added %.sql/run to run postprocess.sql, etc. separately from the install targets they are a part of

schemas/functions.sql: Added to_global_col_names()

schemas/functions.sql: col_global_names(): Use new functions.ensure_prefix() to only add the table name prefix if it doesn't already exist. This makes the function idempotent.

schemas/functions.sql: Added ensure_prefix()

schemas/functions.sql: Added has_prefix()

schemas/functions.sql: Added col_global_names()

schemas/functions.sql: Added name(regtype)

schemas/functions.sql: Added col_names()

root Makefile: Installation: Fixed bug where need to run schemas/public/install separately because schemas/install installs only the util schemas

root Makefile: Installation: install util schemas (temp functions py_functions) before inputs, so that inputs can use util functions in their postprocess.sql or create.sql scripts. (However, they must not use util functions in views or index functions, because these would be cascadingly deleted whenever the util schemas are reinstalled before an import.)

README.TXT: Single datasource import: Added by_col=1 to all commands

mappings/VegCore-VegBIEN.csv: locationRemarks: Remapped to locationnarrative because location.notespublic is a boolean field

lib/sql_io.py: mk_errors_table(): Create a unique index on the MD5 of the value and error instead of on the values directly, because some strings are too long to index (e.g. row 2537268 of MO.Specimen causes an error "index row size 3032 exceeds maximum 2712 for index [...] Values larger than 1/3 of a buffer page cannot be indexed")

inputs/import.stats.xls: Updated import times

bin/map: No mappings warning: Added explanation that this could also be due to no column name matches, and hint to check if you are importing the correct input table

inputs/MO/: Renamed Specimen.2/ -> now available Specimen/

inputs/MO/: Removed old import in Specimen/

Refreshed MO

csvs.py: TsvReader.next(): Fixed bug where empty line needs to be separately returned as [], because csv.reader would interpret it as EOF since the line ending has already been removed

csvs.py: sniff(): TSVs: Turn off quoting because TSVs use \-escapes instead of quotes to escape delimeters, newlines, etc.

csvs.py: InputRewriter.readline(): Surround function in a try block that prints all exceptions, so that debugging information is available if an error occurs when this stream is used as input for psycopg's copy_expert() (COPY FROM)

Populated inputs/MO/import_order.txt

Refreshed SALVIAS

Added web/main/CTFS/

inputs/SALVIAS/: Regenerated salvias_*.schema.sql from the MySQL version, to take advantage of my2pg improvements. The placeholder *_index columns which take the place of MySQL's inline index definitions have now been replaced by no-op CHECK constraints, so that there are no longer lots of dummy *_index columns in the map spreadsheets.

Added web/main/Redmine/ alias to VegBIEN/Redmine/

Added web/main/VegBIEN/Redmine/

web/main/VegBIEN/.htaccess: Forward to new db/ subdir

Added web/main/VegBIEN/db/

web/main/**/.htaccess: Removed RewriteCond -l tests because one of the -d or -f tests will always also pass, making the -l test unnecessary

web/main.conf: Added tolower RewriteMap

web/main/.htaccess: use separate lowercase version when available: Also support input strings in mixed case which is not the default capitalization, in addition to all-lowercase strings

web/main/.htaccess: use separate lowercase version when available: Generate the new dirname with a separate RewriteCond so its value can be used both in the -d test and in the replacement string, rather than separately for each

web/main/.htaccess: translate dotpaths: Allow an unescaped . at the beginning of a filename, because this will never be a . separator. This adds support for hidden files in dir paths, which now won't be interpreted as dotpaths. However, regular files with extensions still need to have the filename escaped because it will otherwise be interpreted as a dotpath.

web/main/.htaccess: Set Options +FollowSymLinks. It should be on by default ("All options except for MultiViews. This is the default setting." <http://httpd.apache.org/docs/2.2/mod/core.html#options&gt;), but this makes sure it will always be enabled.

web/main/.htaccess: Name the lowercased versions of dirs with a leading . (to make them hidden) instead of a trailing _ , to avoid having each dir listed twice in a row in the dir index

Added web/main/TNRS/

Added web/main/VegBank/

Added web/main/BIEN2/

web/main/index.php: Replaced - with . in namespaces to conform to new dotpath naming convention, which allows nesting of namespaces

web/main/SALVIAS/.htaccess: Forward to new dd/ subdir

Added web/main/SALVIAS/dd/

web/main/DwC/.htaccess: Forward to new terms/ subdir

Added web/main/DwC/terms/

web/main/**/.htaccess: don't redirect subdir paths: Fixed bug where can only match non-empty string, because otherwise the rule would match this directory, which should still have its redirects processed

web/main/index.php: Added back smaller spacing between the table columns

web/main/main.css: blockquote: Removed right margin so there isn't a big space between the table columns in index.php, which results from nesting right-padded blockquotes inside one another

web/main/index.php: Changed Brad-Boyle to just Brad because people's names only have to be unique within VegPath

web/main/.htaccess: Added fallback redirect to VegCore for paths without a namespace. This can be used to link to specific VegCore terms without needing to include the VegCore namespace.

Added web/main/VegBIEN/

Added web/main/servers/vegbiendev/

Added web/main/.phpPgAdmin/

web/main/.phpMyAdmin/.htaccess: Set [redirect] flag in case the dest server is on the same machine as VegPath itself

web/main/*/ lowercase versions: Renamed with _ suffix to avoid svn conflicts on case-insensitive filesystems such as Mac HFS+

web/main/.htaccess: Support lowercase versions of mixed-case dirnames without breaking case-insensitive filesystems such as Mac HFS+

web/main/: Added lowercase symlinks for mixed-case dirs to work with subdomain translation, which uses subdomains lowercased by DNS

web/main/index.php: Use absolute URLs for dependencies to work with subdomain translation, which adds components to the URL path

web/main/**/.htaccess: Use RewriteRule instead of RedirectMatch to handle incremental redirects internally instead of issuing a (much slower) redirect to the web browser each time. This also handles edge cases better, as [last] RewriteRules can be used to control when to forward control to a subdir, and doesn't require prepending the path to the dir the .htaccess file is in. Note that this requires all gateway dirs (dirs with subdirs) to contain special RewriteRules to avoid redirecting subdir paths and handle DirectoryIndex; see web/main/DwC/.htaccess. Also note that the regexp of a catch-all RewriteRule must exactly follow the template for internal or external redirects; see web/main/SALVIAS/db/.htaccess for internal redirects and web/main/DwC/history/.htaccess for external redirects.

web/main/.htaccess: translate dotpaths: Allow the part before the [] escape to contain [], to support labels that end in [] (like PHP array vars in the query string) labels with a simple array-subscript syntax (a[b]). This also shortens the regexp and makes it more readable without the \[\] in [^.\[\]/] . Note that this also allows invalid combinations of [] exprs (e.g. more than one per level or unbalanced []), which will still be translated but will probably not have the desired result.

web/main/.htaccess: translate dotpaths: Inline the [] escape regexp into the main regexp, because it is now approximately the same length as the []-matching portion of the main regexp and this greatly simplifies the code by removing the extra RewriteCond. Note that the translation rule is now a plain regexp (run repeatedly until no match), which can be used in any programming language that supports Perl-compatible regular expressions, not just mod_rewrite.

web/main/.htaccess: translate dotpaths: discardpath explanation: Clarified that the infinite loop resulted from reappending PATH_INFO (the Apache-matched filename)

web/main/.htaccess: translate dotpaths: Require any [] escape to have the ] at the end of the level, to simplify the [] regexp

web/main/.htaccess: translate dotpaths: Use a lookahead assertion to ensure that at least one character is matched as the head of the dotpath. This ensures that (.*/)? + the rest of the regexp does not match a path with a trailing /, which is a sealed /-path and not subject to dotpath translation.

web/main/.htaccess: translate dotpaths: Only support one [] escape per dot-level to (greatly) shorten the [] regexp. This does not pose a problem for encoding . because the entire level can simply be enclosed in [].

web/main/.htaccess: Uncommented ErrorDocument