Project

General

Profile

Statistics
| Revision:
  • svn:ignore: *

# Date Author Comment
13947 07/09/2014 02:50 PM Aaron Marcuse-Kubitza

fix: mappings/VegCore-VegBIEN.csv: mapped DwC year/month/day, which are split-date alternatives to eventDate. this fixes the missing eventDate in FIA.

11970 01/20/2014 11:33 AM Aaron Marcuse-Kubitza

moved everything into /trunk/ to create the standard svn layout, for use with tools that require this (eg. git-svn). IMPORTANT: do NOT do an `svn up`. instead, re-use your working copy's existing files with `svn switch` (http://svnbook.red-bean.com/en/1.6/svn.ref.svn.c.switch.html).

11396 10/21/2013 07:14 PM Aaron Marcuse-Kubitza

fix: bin/map: put template: comment out the "Put template:" label so that the output is valid XML, and displays properly in a browser rather than showing a syntax error

10947 09/14/2013 04:34 PM Aaron Marcuse-Kubitza

inputs/FIA/*/map.csv: use -- to separate the table and column name instead of - , to conform with the u-name format (wiki.vegpath.org/u-name#format), which works even when only one of _- can be used in the name. -s are needed in this case to linewrap the column on a separate line as the table in phpPgAdmin.

10866 09/04/2013 11:06 PM Aaron Marcuse-Kubitza

inputs/*/*/test.xml.ref: updated source.shortname for new datasource name, which now starts out with .new suffix

10174 07/06/2013 03:55 PM Aaron Marcuse-Kubitza

bugfix: inputs/input.Makefile: %/VegBIEN.csv: for new-style datasources, use a symlink to mappings/VegCore-VegBIEN.csv directly instead of prefiltering VegCore-VegBIEN.csv to include only the columns in map.csv. prefiltering used to be performed as part of mapping the map.csv VegCore output terms to VegBIEN using bin/join, but is no longer needed because the staging table columns are now VegCore terms. instead, the full VegCore-VegBIEN.csv is needed so that derived columns added in stage I or II validations are detected by bin/map (rather than just the original source columns in map.csv).

10090 06/27/2013 12:23 PM Aaron Marcuse-Kubitza

added inputs/FIA/*/{VegBIEN.csv,test.xml.ref}, which are now generated by the mapping process for the joined-together tables (even though they are not used by the import, because only occurrence_all is imported)

10086 06/27/2013 12:12 PM Aaron Marcuse-Kubitza

inputs/FIA/*/postprocess.sql: removed svn:executable attribute using `svn pdel svn:executable ...` now that these are not shell scripts

10084 06/27/2013 12:10 PM Aaron Marcuse-Kubitza

inputs/FIA/*/import: changed to postprocess.sql for use by the runscripts

10082 06/27/2013 04:26 AM Aaron Marcuse-Kubitza

added inputs/FIA/*/run. these do not yet use the postprocessing operations in */import.

10054 06/26/2013 11:05 AM Aaron Marcuse-Kubitza

inputs/FIA/SUBPLOT/map.csv, import: prepended * to all FIA terms to clearly distinguish them from the VegCore terms. this is the standard convention for all datasources, to indicate which terms have not yet been mapped, but was not yet implemented at the beginning of new-style import (the FIA refresh was the first new-style datasource)....

10052 06/26/2013 08:58 AM Aaron Marcuse-Kubitza

added inputs/FIA/*/_no_import to src tables that are joined together in occurrence_all and should not also be imported separately once they are in import_order.txt

8801 05/02/2013 08:53 PM Aaron Marcuse-Kubitza

inputs/input.Makefile: SVN: add, %/add: */logs: also svn:ignore *.gz, used for compressed log files

8316 04/04/2013 01:16 AM Aaron Marcuse-Kubitza

inputs/FIA/*/map.csv: Changed newlines between table and field name to - because the newlines mess up the flow of queries and also break pgAdmin's display of EXPLAIN output. The - was chosen because it's a non-whitespace character that linewraps in browsers, phpPgAdmin, and Google spreadsheets (although unfortunately not in pgAdmin). It is better than space because you can set a text editor to treat it as a word character, allowing the entire column name (<table>-<field>) to be selected by double-clicking it.

8240 03/28/2013 05:21 PM Aaron Marcuse-Kubitza

inputs/FIA/: Ran inputs/FIA/import. This maps to VegCore's commonName.

8237 03/28/2013 04:59 PM Aaron Marcuse-Kubitza

inputs/FIA/PLOT/map.csv: ELEV: Remapped to elevation_ft, assuming units based on the actual elevation of the region for a sample plot record

8208 03/27/2013 08:35 PM Aaron Marcuse-Kubitza

inputs/FIA/*/import: Run mk_derived after postprocessing commands

8184 03/27/2013 12:54 PM Aaron Marcuse-Kubitza

inputs/FIA/*/import: Removed util. before function names because util is in the search_path

8183 03/27/2013 12:43 PM Aaron Marcuse-Kubitza

schemas/*functions.sql: Renamed to *util.sql because now that these schemas are used by the new-style import scripts, there can be more than just functions in them

8177 03/25/2013 09:45 PM Aaron Marcuse-Kubitza

inputs/FIA/*/map.csv: Replaced . between table and column name with newline, so that table viewers like pgAdmin will display both the table and column name at the left edge of the header cell, rather than displaying only the table name because the column name doesn't fit. This fixes the problem of seeing a bunch of columns whose names all start with a table name, and not knowing what each of them is. It also preserves the ability to see at a glance which table a column is in, which helps in navigating wide tables. Removed * before unmapped terms, because whether a term is mapped is generally obvious from the table name itself.

8176 03/25/2013 09:01 PM Aaron Marcuse-Kubitza

inputs/input.Makefile: %/.map.csv.last_cleanup: Run fix_line_endings after canon/translate to standardize Python's \r\n line endings back to \n. This prevents issues with mixed line endings because LibreOffice (and probably Excel) treat all cell-internal line endings as \n but row line endings as whatever the file had, while text editors like jEdit translate all line endings to whatever the autodetected line ending is. (This creates spurious line ending diffs when a map spreadsheet containing multiline cells is edited in a text editor.)

8172 03/25/2013 07:38 PM Aaron Marcuse-Kubitza

inputs/FIA/*/map.csv: Mapped terms to VegCore

8170 03/25/2013 06:57 PM Aaron Marcuse-Kubitza

inputs/FIA/*/import: Updated column names to match map.csv

8166 03/25/2013 04:55 PM Aaron Marcuse-Kubitza

inputs/FIA/*/map.csv: Removed no longer needed leading . from joined fields (globally-unique terms), because functions.to_global_col_names() is not used anymore

8164 03/25/2013 04:36 PM Aaron Marcuse-Kubitza

inputs/FIA/*/import: Use map_table to set column names based on the contents of map.csv, instead of using functions.to_global_col_names() and functions.rename_if_exists(). Added map.csv for all tables.

8163 03/25/2013 03:19 PM Aaron Marcuse-Kubitza

inputs/FIA/: Changed postprocess.sql scripts to import scripts that can be run directly. Added top-level inputs/FIA/import to run all of them together.

8136 03/21/2013 06:01 AM Aaron Marcuse-Kubitza

inputs/FIA/*/postprocess.sql: Avoid using :table, :table_str so that the commands in the script can also be run by pasting them into pgAdmin

8130 03/21/2013 01:27 AM Aaron Marcuse-Kubitza

inputs/FIA/*/postprocess.sql: Added index on *.CN (autogen IDs)

8111 03/20/2013 08:52 AM Aaron Marcuse-Kubitza

inputs/FIA/*/postprocess.sql: Cluster tables by their *.unique index for faster joins

8110 03/20/2013 08:51 AM Aaron Marcuse-Kubitza

inputs/FIA/*/postprocess.sql: Cast ID columns to integer using new functions.set_col_types()

8098 03/20/2013 03:00 AM Aaron Marcuse-Kubitza

inputs/FIA/*/postprocess.sql: Removed table prefix from globally-unique columns that should be joined on

8093 03/20/2013 01:28 AM Aaron Marcuse-Kubitza

inputs/FIA/*/postprocess.sql: Use functions.to_global_col_names() to ensure that all column names are globally unique. This makes it easy to join the tables together without worrying about column name collisions.

8092 03/20/2013 01:15 AM Aaron Marcuse-Kubitza

inputs/FIA/*/postprocess.sql: Use new functions.create_if_not_exists() to allow re-running postprocess.sql idempotently

7793 02/28/2013 11:26 AM Aaron Marcuse-Kubitza

inputs/FIA/*/postprocess.sql: UNIQUE constraint: Added UNITCD because although it may not be needed to scope PLOT, it is present in all the core tables and thus must be part of the joins' USING columns

7792 02/28/2013 10:58 AM Aaron Marcuse-Kubitza

inputs/FIA/*/postprocess.sql for core tables: Added unique constraint on the datasource's primary key fields

7387 01/29/2013 03:26 PM Aaron Marcuse-Kubitza

inputs/FIA/: Added refreshed data (not yet mapped)