Python Developer with Unix cron & AWS
Sunnyvale, CA
Job Description:
JobDescription:
Minimum 9+ years of softwareprogramming experience
Strongexperience in ETL
SQLscripting experience
StrongPython development experience around data management.
Strongin Unix cron not just exposure
Musthave the ability to work in a dynamic, fast-paced environment
Strongcommunication skills to interact with Agile team members
DataSources: Cloudbased as well as on premise example: SAP CRM, ECC, people soft, PLM,Calliduscloud
High-levelarchitecture:
Amazon S3 bucket have foldersfor specific source files delivered by Fusion middleware. Data quality check isnot handled by the sources.
AnacondaPython based programs are in place to clean any abnormalities in the sourcefiles and re-write them back in their respective folders this is to avoid loadfailures.
mostof the known errors are handled using this mechanism, moving forward you willrequire to enhance these scripts to include a fix for any new errors.
currentlymechanism is processing one file at a time, target is to move it to parallelprocessing.
Nextis a Python program for data pipeline, based on AWS API and snowflake API. Thisis towrite files on snowflake EDW
Thereare 3 different layers on the snowflake side Pre=processing, staging and basetables
Unixcron is used for scheduling purpose.