Versions Compared

Old Version 1

changes.mady.by.user Arielle Kaye Odeste

Saved on

compared with

New Version Current

changes.mady.by.user Arielle Kaye Odeste

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Table of Contents

...

Overview

CSV adapter underwent dramatic changes in terms of design patterns, code clean up by removing code duplication, use of common API and creating new nugets and repos.

It also features the refactored timezone implementation.

...

Product Enhancements

  • CSV adapter date time refactor - Datetime parsing is not working when timezone of the server machine is not UTC. Workaround would be changing the server timezone to UTC. Datetime parsing has been a recurrent issue due to CSV files coming up with multiple date formats or different timezones. Refactoring is needed to address the current logic of handling date time and its configurations.

  • RFE-161 CSV Adapter - Timezone Implementation - Currently, the CSV files should be in UTC in order to index the data in UTC in ES. We should be able to handle if the data in the file is not in UTC and at a different timezone.
    New adapter configurations: SourceTimeZone, and DestinationTimeZone

  • CSV Adapter DateTime to UTC

  • Update metadata when column is removed - Adapter should always work based on metadata only. That when a column is suddenly added or removed, the adapter should be able to handle it correctly and still onboard according to metadata.

  • RFE-162 Handling Additional Column Insertion in the CSV file - CSVMetaData is the one that dictates which columns will be onboarded on the index.
    In scenarios when source CSV file has an additional column that is not listed in the CSVMetaData, the adapter will ignore the additional column and will be able to proceed with the onboarding.

  • RFE-162 Handling Deleted Column Removal in the CSV file - CSVMetaData is the one that dictates which columns will be onboarded on the index.
    In scenarios when source CSV file has a missing column that is listed in the CSVMetaData, the adapter will still display the missing column in the index but will be null since no such columns are in the source CSV file. The adapter will still be able to proceed with the onboarding.

  • RFE-163 CSV Adapter - Column order in the CSV file should not impact the data loading. This is to handle switching column orders in CSV Metadata or in CSV File.

  • Upgrade the CSV adapter to start using the new AWS Authentication and S3 nugets

  • Move out SFTP helper of CSV Adapter to a nuget

  • Move CSV Common Api to Nuget

  • Upgrade the CSV and Excel Adapter to start using the new eMiteSDK.SFTP nuget - As part of decoupling the CSV Adapter, it needs to use new nugets (AWS Common API, Authentication, and S3 Helper) instead of directly referencing the dependency project/s.

  • [CSV] Error in logs when MaskFields config is missing - Fixed PI masking logging errors by removing PI masking functionality. The adapter should be able to handle even if the maskfields config is missing from the instance or it contains empty string.

  • ESD-5390 CSV File Parsing error - Upon replacing the path with a simpler one and keeping the CSV file in a simple folder that does not use brackets or special letters in the folder name, the file was able to be read and processed without any issue.