inputs/CTFS/test: Accepted initial test outputs. A lot of leaves are still unmapped with the default mappings.
inputs/CTFS/maps: Added initial maps
VegX mappings: taxonConcept mappings: Added "tcs:" namespace prefix to appropriate elements. This will make the taxonConcept XPaths compatible with CTFS VegX.
input.Makefile: Maps building: full via maps (maps/$(via).%.full.csv): $(makeFullCsv): Sort all maps so that rows are re-ordered whether or not a core self map exists. This way, if a core self map is created, it will not cause the sort order of the generated via-format XMLs to change. This makes it easier to accept any changes to test outputs that result from adding a core self map.
mappings/Makefile: VegX: Added VegX.self.organisms.csv. Added root attrs to chRoot maps, commented out since it's not ready to be checked in yet.
xpath.py: get(): Run xml_dom.by_tag_name() with ignore_namespace=False (possibly later set to True)
xml_dom.py: Comments: Added clean_comment() and mk_comment(). Searching child nodes: by_tag_name(): Added ignore_namespace option to ignore namespace of node name.
root Makefile: Added %-remake target
mappings/Makefile: Renamed joinMaps to dwcMaps and chrootMaps to vegxMaps. Added commented-out code to create VegX.self.organisms.csv (not ready to check in yet because it affects many dependent maps).
input.Makefile: Removed no longer needed $(noEmptyMap)
xml_func.py: process(): Use new xml_dom.mk_comment()
xml_dom.py: Added clean_comment() and mk_comment() to properly sanitize comment contents (comments can't contain '--')
Added inputs/TRTE
inputs/QMOR/test: Added initial accepted test outputs
inputs/QMOR/maps: Added maps
Added inputs/QMOR
inputs/MT/test: Added initial accepted test outputs
inputs/MT/maps: Added maps
mappings/Makefile: DwC-VegBIEN.specimens.csv: Don't call remove_empty to produce it, because join now deals with empty mappings correctly by still raising a warning. Removed no longer needed intermediate DwC.ci-VegBIEN.specimens.csv.
join: Also print "No join mapping" warning if a join mapping was found but it was empty. The warning in that case is actually "No non-empty join mapping" to distinguish it from a mapping that's missing entirely. input.Makefile: missing_mappings: Support new "No join mapping" error message.
Added inputs/MT
Added disown_all to disown all running jobs
stop_imports: Call jobspecs relative to $selfDir, rather than assuming it will be run from the svn root dir
union: Call maps.merge_headers() using **dict(prefer=header_num) instead of just prefer=header_num in order to work on Python 2.5.2 (which nimoy is running)
inputs/ACAD/test: Accepted initial test outputs
Added inputs/ACAD/maps/ maps
Accepted new test outputs resulting from the addition of the id -> occurrenceID mapping in mappings/DwC1-DwC2.specimens.csv
inputs/SALVIAS*/maps: Cleaned up maps for the first time since all via maps became subject to cleanup
input.Makefile: Removed no longer needed default "maps/.$(via).%.csv.last_cleanup" rule
input.Makefile: Maps building: Via maps cleanup: Added `env ignore=1` since with the switch to subtracting $(coreMap), all inputs will attempt to subtract some map, even if it's not subtractable
input.Makefile: Don't clean src maps, only build them
inputs/ARIZ/maps/DwC.specimens.csv: Re-cleaned up to take advantage of additional entries now removed by subtract
input.Makefile: Maps building: Via maps cleanup: Subtract $(coreMap) instead of $(coreSelfMap) so that entries whose input and output maps to the same place are subtracted as well
subtract: Also remove mappings whose input and output maps to the same non-empty value in map_1
util.py: Added all_equal(), all_equal_ignore_none(), have_same_value()
mappings/DwC1-DwC2.specimens.csv: Added id -> occurrenceID mapping
inputs/SALVIAS-CSV/maps/VegX.%.full.csv: Regenerated using new src maps
mappings/DwC1-DwC2.specimens.csv: Added mappings from dcterms elements without namespace to with namespace
inputs/SALVIAS-CSV: Built maps/src.%.csv
Added inputs/ACAD/maps/src.specimens.csv
input.Makefile: Maps building: Autogen src maps with known table names. Sources: $(withCatSrcs): Fixed bug where substitution pattern did not contain %.
Added src_map to make a source map spreadsheet from a CSV header
input.Makefile: Split Maps section into "Existing maps discovery" and "Maps building" sections. Sources: Added cat, cat-% to cat out sources.
input.Makefile: Factored out sources-related code to new Sources section
input.Makefile: $(srcMaps): Removed `$(filter-out maps/src.join.%.csv,...)` because maps/src.join.%.csv are no longer created
README.TXT: Schema changes: Split updating graphical ERD exports into separate section. Update graphical ERD exports: Added schemas/vegbien.ERD.core.pdf .
README.TXT: Added Datasource setup section with instructions to add a new datasource
Added inputs/ACAD
input.Makefile: Only setSvnIgnore the input dir, since it already exists and doesn't need to be added (inputs/Makefile adds it)
inputs/*/maps/DwC.specimens.csv: Removed extranenous XML meta info from DwC column root, since it now just needs to be present in the core via map mappings/DwC-VegBIEN.specimens.csv
union: Use new maps.merge_headers() to write properly combined header
maps.py: join_combinable(): Fixed roots_combinable() to run on col names instead of roots, which were passed in. merge_mappings(): Factored out mapping column combining into merge_mapping_cols(), which handles an optional prefer param as well to take the header_num env var. Added merge_headers().
util.py: Added sort_by_len(), shortest(), longest()
join: Use new maps.join_combinable() to check if column names match
maps.py: Added cols_combinable() and use it in combinable(). Added join_combinable() and associates helper functions. Added documentation labels to each section.
xml_parse.py: ConsecXmlInputStream: Removed read() because that's now defined in streams.FilterStream
xml_parse.py: parse_next(): Strip control characters from input stream because they mess up the parser
streams.py: FilterStream: Forward all reads to readline()
strings.py: Added is_ctrl() and strip_ctrl()
xml_parse.py: parse_next(): On parser error, advance to next XML document since the rest of the current document is corrupted
streams.py: Added consume(). Added documentation labels to each section.
bin/map: For XML inputs, wrap sys.stdin in a LineCountStream and use new xml_parse.docs_iter() on_error() to add input line # to XML parsing exceptions
xml_parse.py: Added on_error() handler to parse_next() (passed through by docs_iter()), so that the caller can add useful info like the input line # to the exception message, and decide not to suppress rather than re-raising the exception
VegX-VegBIEN.organisms.csv: Renamed individualOrganismObservation user-defined field identificationLabel2 to identificationLabel. Distinguish what are now two identificationLabel fields of the same name by tagging each one with [@id=2] or [@id=1]. inputs/SALVIAS-CSV/maps/VegX.organisms.csv: Merge tag1/stem_tag1 and tag2/stem_tag2 using _alt, since they are never set to different values when both are not NULL (although sometimes just one or just the other is not NULL).
VegX-VegBIEN.organisms.csv: Renamed individualOrganismObservation user-defined field tag2 to identificationLabel2 to reflect that it will become a second instance of identificationLabel
VegX-VegBIEN.organisms.csv: Re-mapped individualOrganismObservation user-defined field lineCover to already existing volumeCanopy
VegX-VegBIEN.organisms.csv: Re-mapped individualOrganismObservation user-defined field cover to already existing attribute.coverPercent
VegX-VegBIEN.organisms.csv: Re-mapped individualOrganismObservation user-defined field count to already existing aggregateOrganismObservation.aggregateValue
vegbien.ERD.mwb: Fixed lines
README.TXT: Documented that `make reinstall_db` will delete your VegBIEN DB
README.TXT: Documented that `make empty_db` will delete your VegBIEN DB
root Makefile: empty_db: Confirm deletion just like for rm_db. rm_db: put $(confirmRmDb) on a separate line and move the $(error) call to the main $(confirm) macro since you always want to abort make if the user cancels (not just not run that command).
root Makefile: rm_db: If user cancels, abort in case target was reinstall_db to prevent installing
root Makefile: core, rm_core: Fixed bug where no longer existing prerequisites postgres_user, rm_postgres_user were not removed
root Makefile: rm_db: Confirm deletion with user. Merged postgres_user, rm_postgres_user into db, rm_db so that deletion confirmation applies to user deletion as well (which would indirectly cause the DB to be deleted).
README.TXT: Testing: Updated to add missing mappings
root Makefile: test-all: Added missing_mappings
Moved maps validation targets from main Makefile to input.Makefile. main Makefile: maps validation: Summarize the output of the inputs' maps validations.
Makefile: Also find missing input mappings, in addition to missing join mappings
join: Also produce warnings for no input mapping (if no comment explaining why no input mapping), in addition to no join mapping
inputs/NY/maps/DwC.specimens.csv: Documented why there is no input mapping for key
VegX-VegBIEN.organisms.csv: Renamed individualOrganismObservation user-defined fields stem* to remove the stem* prefix to be consistent with VegBIEN
VegX-VegBIEN.organisms.csv: Renamed individualOrganismObservation/plotObservation user-defined fields sourceaccessioncode to sourceAccessionCode to be consistent with VegX case sensitivity
VegX-VegBIEN.organisms.csv: Renamed individualOrganismObservation user-defined field interceptCm to lineCover to be consistent with VegBIEN
VegX-VegBIEN.organisms.csv: Renamed individualOrganismObservation user-defined field individualCode to authorPlantCode to be consistent with VegBIEN
VegX-VegBIEN.organisms.csv: Renamed individualOrganismObservation user-defined field htFirstBranchM to heightFirstBranch to be consistent with VegBIEN
VegX-VegBIEN.organisms.csv: Renamed individualOrganismObservation user-defined field coverPercent to cover to be consistent with VegBIEN
VegX-VegBIEN.organisms.csv: Renamed abioticObservation user-defined field siltPercent to silt to be consistent with VegBIEN
VegX-VegBIEN.organisms.csv: Renamed abioticObservation user-defined field sandPercent to sand to be consistent with VegBIEN
VegX-VegBIEN.organisms.csv: Renamed abioticObservation user-defined field pottasium to potassium to be consistent with VegBIEN
VegX-VegBIEN.organisms.csv: Renamed abioticObservation user-defined field organicPercent to organic to be consistent with VegBIEN
VegX-VegBIEN.organisms.csv: Renamed abioticObservation user-defined field clayPercent to clay to be consistent with VegBIEN
VegX-VegBIEN.organisms.csv: Renamed abioticObservation user-defined field cationCap to cationExchangeCapacity to be consistent with VegBIEN
VegX-VegBIEN.organisms.csv: Renamed plotObservation user-defined field precipMm to precipitation to be consistent with VegBIEN
VegX-VegBIEN.organisms.csv: Changed plotObservation user-defined field plotMethodology to /simpleUserdefined[name=method]/*ID/method/name
schemas/postgresql.nimoy.conf: Increased default_statistics_target to 8.4 default value to improve execution query plans
Added schemas/postgresql.Mac.conf (for tuning developers' local testing DBs)
schemas/postgresql*.conf: Increased checkpoint_segments and checkpoint_completion_target so that checkpoints (performance intensive) are written less often and load-balanced better
