There was a time - now long gone - when data teams were mostly concerned with generating PowerPoint decks and interpretive models for optimizing business strategy. What pressures are data leaders under today? Indeed, data leaders and their teams sometimes even have to act as disruptors, challenging the way businesses are run based on new insights. So data leaders have to be prepared to take projects in a different direction from what the business wanted. It may not even give clear answers to the questions we’ve posed, but instead suggest new questions. They should recognize valuable insights that emerge from data (above and beyond the insights that the business already demands)ĭata teams don’t know what the data is going to tell them until they look at it.It’s challenging enough just to deliver reliable metrics, but it’s even more of a challenge to deliver predictive models that yield accurate predictions - in an ethical and timely manner - directly into end-user applications! They have to create operational analytics and models that change the way the company does businessĭata leadership is not about static insights and static metrics anymore, but about production models that plug directly into applications to improve people’s day-to-day work and enhance the end-user experience.But beyond that, you need to get out of the way and empower each department to create its own metrics that meet those same standards. You need consistent definitions of basic business metrics, and you need to make sure that those are well documented, accurate, and reliable. This is the bread and butter of a data team, of course, and it’s still surprisingly hard to do well. They need to deliver accurate metrics in a reliable manner.What are the main goals of data and analytics leaders? It enables us to obtain the right data at the right time and in the right format, and then push the insights and models into our lines of business.Īdditionally, as frontline users of Airflow within Astronomer, we provide information about the platform to our Product team, such as what opportunities and challenges exist, as well as how we can improve the project for ourselves and the community. Astronomer doesn’t just offer a self-hosted and SaaS-managed Airflow with commercial support - we rely entirely on Airflow ourselves to manage all our data pipelines and analytics. To be able to do this, we turned to the same modern data orchestration platform that Astronomer offers to customers. For example, we tell the support engineers how quickly organizations are upgrading to the latest version of Airflow, and whether they’re running into any problems - information that lets them deliver more value to customers. My data team analyzes data generated by the Astronomer and Airflow ecosystems, to provide business insights that directly impact how other Astronomer teams do their work. What does your data team do, and how does it do it? To talk about data orchestration with Airflow from the perspective of a data leader, we reached out to someone intimately familiar with both the role and what Airflow can do: Steven Hillion, Astronomer’s own VP of Data. Our goal is to provide all of them with the most frictionless experience of Apache Airflow possible, which means understanding all their varied needs, and especially the needs of their team leaders. At Astronomer, we serve a wide range of data practitioners, including data engineers, data architects, machine learning engineers, business analysts, and data scientists.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |