Using a simple Airbyte job not only eliminates the need for manually passing CSV files between clients and accountants but also ensures that your data stays in sync automatically.

Developing and managing pre-configured Airbyte connectors with Terraform removes the dependency on DevOps.

Beware: Neglecting the importance of correct timestamp format in incremental sync can lead to data chaos, system errors, and operational disruptions!

Airbyte's open-source nature aligns seamlessly with our values of commitment to transparency, flexibility, and community-driven innovation.

Developers have been tracking dependent API changes for years, we need to include data engineering as schema consumers that also need to be made aware of changes.

Deleting data is tough and feels like a big responsibility. The best way to avoid hoarding is to not gather unnecessary data in the first place.

Too many models, exasperating tests, unreadable code and poor performance all hamper DBT projects

Any workload in the Cloud can be compromised through access to raw memory via the supervisor

Airbyte is the Swiss Army knife for ETL tasks

Snowflake is just better at big data than Redshift or Postgres RDS. Snowflake's justified success will continue.

A jack of all trades is a master of none, but oftentimes better than a master of one.

Utilizing a combination of schema validation for structured data, regular expressions to search for specific items, and AI for text and document analysis will yield fast, cost-efficient, and reliable results when removing Personally Identifiable Information (PII) from data.

DO NOT IGNORE IT! This type of activity, hidden from the eyes of peers, is usually a ticking bomb waiting to explode

The NHS haven't chosen to go with Palantir. A controversial one given the involvement with contrarian billionaire, Peter Thiel, but in defence of NHS procurement, they are best placed to make progress.

Adopting low-code tools and platforms for Data Management offers clear advantages such as ease of adoption, a low entry level for non-technical personnel, and, of course, a variety of integrations provided out of the box.

There is more to user needs data outside of web analytics, data engineering challenges organisations to tackle this

iPaaS transforms data extraction into a seamless and secure operation, simplifying the process, ensuring repeatability, and fostering reliability in modern integration landscapes.

How does one make sense of the wast amounts of data availiable and ensures it’s accessible, searchable, and usable? Enter metadata. Creating metadata for each dataset is essential, and a data catalogue acts as an index and search function for this metadata.

An iPaas is a great option to cut down integration time and introduce agility to your data organisation. This field is destined to grow and new players will emerge.

Companies that have bought into the light-touch workflow approach are inevitably going to be in trouble. There is a ticking technical debt clock that sits in your code that will need to be mitigated.

Applied Observability is the utilisation of observable data in a cohesive and holistic manner across various business functions. It differs from Observability in that combines multiple Data Layers present in the organisation to shorten the time between stakeholder actions and organizational reactions enabling proactive planning of business decisions.

Camunda’s expertise in process automation offers more than meets the eye – it holds valuable insights within orchestrated workflows, waiting to be discovered.