In the data world, quality is everything. A machine learning model trained with incorrect data, a dashboard showing wrong information, or a report making decisions based on incomplete data can have devastating consequences, from financial losses to irreparable reputational damage. Great Expectations was born to solve this problem, offering a systematic, automated and collaborative framework to ensure that data meets the quality standards defined by the organization at every stage of the data pipeline.
Great Expectations (GX) is an open source framework written in Python that allows you to define, document and execute automated and reproducible data quality tests. It was created in 2017 by Great Expectations and has since become the de facto standard for Data Quality as Code, being adopted by thousands of organizations worldwide, from startups to Fortune 500 companies. Its focus on quality as an integral part of data pipeline development, rather than a subsequent step, has revolutionized the way data teams ensure the reliability of their information.
GX philosophy is simple but powerful: your data must meet a series of expectations (quality rules) that you define. If they don't, the pipeline fails and the team is immediately notified, enabling quick response and correction of issues before incorrect data reaches consumers. But GX also generates beautiful, navigable HTML documentation that serves as a living data quality catalog, providing a clear and up-to-date view of data health at all times, facilitating auditing and communication of data quality to business stakeholders.
There are many data quality tools, but GX stands out for several fundamental reasons. It is open source and free, with no license costs and an active community and continuous development that ensures the evolution and constant improvement of the tool. It offers over 50 predefined expectations, from basic validations like not_null and unique to advanced ones like statistical distributions, accepted values and regex patterns, covering most common use cases. Automatic documentation is one of its most differentiating features, generating an HTML site with the quality status of all your datasets that can be shared and consulted by the entire team. Native integration with SQL, Spark, pandas, Airflow, dbt and virtually any data ecosystem technology makes GX extremely versatile and easy to incorporate into existing architectures. Finally, it is extensible, allowing you to create custom expectations in Python for specific use cases not covered by predefined expectations.
Installation is easy with pip: pip install great_expectations. To initialize a project, run great_expectations init, which creates the necessary directory structure: great_expectations.yml for global configuration, expectations/ for expectation files in JSON format, checkpoints/ for checkpoint configuration, datasources/ for data source connections, and uncommitted/ for temporary unversioned files. This organized structure facilitates project management and team collaboration.
The fundamental concepts of Great Expectations are the basis of its operation. Expectations are the heart of GX, validation rules you define about your data. Examples include expect_column_values_to_not_be_null("email") to ensure email is not null, expect_column_values_to_be_between("age", 0, 120) to verify age is in a valid range, and expect_column_values_to_be_in_set("country", ["US", "UK", "CA"]) to ensure country is one of the allowed values. Expectations can be simple or complex, and can be combined to create robust validations covering all critical aspects of data.
The Data Context is the central object that manages the entire project configuration, including data location, applicable expectations, and storage of validation results. It provides a unified interface to interact with GX and is the entry point for all tool operations. A Checkpoint is a set of validations that run together, allowing grouping of expectations by table, by pipeline, or by data domain. Checkpoints facilitate executing validations in different contexts and managing quality at different pipeline stages. Data Docs are the automatically generated HTML documentation, including the status of each expectation, validation results and quality metrics, providing a complete and accessible view of data quality for the entire team.
A complete practical example of using GX involves loading data with pandas, converting it to a Great Expectations DataFrame, and defining corresponding expectations. For a customer dataset, this would verify that email and name are not null, age is between 18 and 99, country is one of the allowed ones, email has a valid format via regex, and customer_id is unique. Finally, validation is executed and results are obtained, allowing appropriate action if any expectation fails.
Integration with Apache Airflow is one of the most powerful, allowing validations to be run as part of a DAG. A DataContext can be created and the corresponding checkpoint retrieved to run validations, and if they fail, an exception can be thrown to fail the DAG and notify the team. This integration makes data quality a mandatory step in the data pipeline, ensuring only data meeting quality standards reaches consumers.
If you use dbt, Great Expectations is the perfect complement. dbt already has quality tests, but GX offers much more advanced validations, including statistical analysis and distribution validations. dbt can be used for business tests and transformation logic, and Great Expectations for statistical and general quality validations, achieving total and complementary quality coverage.
The comparison with alternatives shows GX's strengths. Soda is simpler and faster to implement, but less flexible and with fewer features. dbt tests has native integration with dbt but is limited to the dbt ecosystem. Apache Griffin is scalable for big data but complex to configure. Monte Carlo offers complete data observability but is commercial and expensive. Great Expectations positions itself as an open source, flexible option with a large community, ideal for organizations seeking a powerful and extensible data quality tool.
Best practices for Great Expectations include starting small with the most critical tables, automating validation execution by integrating GX into CI/CD pipelines, collaborating by sharing Data Docs with the entire data team and stakeholders, iterating by adding new expectations gradually as new risks are identified, documenting the why of each expectation so it has business meaning, and not being too strict by setting reasonable thresholds, as 100% quality is not always necessary or possible.
Real-world use cases span multiple industries. In e-commerce, product prices are validated to be non-negative and stocks consistent across systems. In banking, transactions are verified to have the correct format and IBANs to be valid. In healthcare, patient data is ensured to be complete and consistent. In marketing, emails are validated to be in the correct format and phones to have the appropriate prefix. In all cases, Great Expectations provides a layer of trust in data that enables more confident decision-making and reduces the risk of errors.
The future of Great Expectations is promising, with the tool in full growth. Version 1.0 was released in 2024 with significant improvements in performance and usability, and the community is working on greater AI integration to automatically generate expectations, better support for streaming and real-time data, and more advanced data observability. These developments position Great Expectations as a fundamental tool in the modern data ecosystem.
Great Expectations is an indispensable tool for any data team that wants to guarantee the quality of their most valuable asset. Its focus on Data Quality as Code and its ability to generate living documentation make it an investment that pays dividends from day one, improving trust in data, reducing issue resolution time, and facilitating collaboration and regulatory compliance.
At Curaduriadedatos.com, we recommend Great Expectations as the first data quality tool any organization should implement. Start today and transform how your team trusts data, building a quality culture that drives the success of your data projects.