Web3 Data Engineer (Office of the CTO)

Data Engineer About ConsenSys Data ConsenSys Data sits within Consensys Software Inc. to help address all our variants of data, break down silos, enable best practices, provide first-rate resources, and accelerate our mission of becoming a cutting-edge data-driven organization. We are using a mix of providing some centralized data engineering functions as a shared service while enabling our business units to make great data decisions with their own data functions. ConsenSys Software Inc. is a wide organization with each individual business unit facing unique data challenges. Infura needs to provide real-time analytics on top of a data pipeline doing billions of events per day. MetaMask Swaps needs to provide financial accounting for a purely on-chain data source. Truffle needs to track developer engagement across the open source ecosystem. We are looking for a data engineer to join our shared data engineering team, with the goal of helping to build, maintain and evolve our data warehouse to support the organization. You will join a team to work with the business to ensure we have a first-class data warehouse supporting our business units and our business decision-making. Some of the key areas you will help ensure we are doing well are: - Data quality - Data governance - Master data management Responsibilities: - Understanding the business and data strategy - Contribute to the collection, storage, management, quality, and protection of data - Implementing data privacy policies and complying with data protection regulations - Effectively communicate the status, value, and importance of data collection to executive members and staff - Knowledge of relevant applications, big data solutions, and tools - Knowledge of real-time streaming data pipelines - Governance: Advising on, monitoring, and governing enterprise data - Operations: Enabling data usability, availability, and efficiency Innovation: Driving enterprise digital transformation innovation, cost reduction, and revenue generation - Analytics: Supporting analytics and reporting on products, customers, operations, and markets Who we’re looking for: - 5+ years of work experience in an enterprise engineering domain - Excellent problem-solving skills and sharp attention to detail - Solid written and verbal communication skills - Proficient in Python programming and well versed with popular frameworks like Pandas, Flask, Airflow, Apache Beam, etc. - Experience with any one cloud technology i.e GCP, AWS, or Azure (GCP preferred) - Hands-on experience with Cloud function, Big Query, DataFlow, Cloud Run, Airflow, etc. - Strong with Linux Commands and Shell Scripting - Experience with Docker & Kubernetes - Experience building CI/CD pipelines - Knowledge of any one SCM tool like Git, BitBucket, etc. - Strong knowledge of Terraform scripting - Strong skills with writing SQL - Enthusiasm for shipping high-quality code and helping peers do the same - Proactiveness and self-driven to be successful working in a remote environment - Understanding of web development practices and terminology - Relevant knowledge of cloud security and data security - A belief in our mission and values Bonus points: - Blockchain expertise - Hands-on experience with Kafka - Hands-on experience with Apache Spark (py-spark preferred) - You’re a MetaMask user!