Tag Archives: cloud

Taming Database Challenges: Insights from Redgate Keynote

Leave a reply

I am excited to cover the Microsoft Keynote on Day 2: Redgate Keynote: Simplifying Complexity – Making the Database Work in the Real World. As the database landscape grows increasingly complex and the pace of change accelerates, robust database practices are essential to manage this complexity effectively. However, fully leveraging the value of databases remains a significant challenge.

In this keynote, Redgate will present real-life stories, insights, and solutions, highlighting both the human and technical challenges associated with databases. We will be joined by a respected industry expert from IDC Europe and a fellow IT leader who is at the forefront of addressing these challenges. This session will feature the latest research, best practice advice, personal anecdotes, and demonstrations of new product offerings designed to help you harness the benefits of mature database practices and unlock the full potential of your data estate.

Updates to follow:

98 sessions, 50 clinic meetings, and a massive amount of networking events.

Day 2 will bring forth more sessions, expo expedition, community zone, community experts clinic and over 70 more sessions starting today.

Women in Technology Luncheon featuring Jes Chapman as the Keynote speaker will be in Ballrooms 2 and 3 over lunch.

Kellyn Gorman takes the stage providing some stats such as 21 % of organizations are using synthetic data for testing.

71% of organizations in the survey were using manual methods to create testing data, which is incredibly time-consuming.

Graham McMillan, CTO at Redgate, talks about all things releases and how the complexity is.

Digital Technology spend will expand seven times faster than the global economy in 2024

Speed (46%), Quality (43%), Efficiency (43%), and Productivity (28%) to help deliver excellence by 4 strategic priorities.

Excessive technical debt forces overspending on infrastructure

Average hours for DBAs to deploy databases is a thing

Time spent on new app functionality is part of slowing business.

Infrastructure environments are changing, bringing forth some additional challenges in today’s world which lends issues to Operational and Governance challenges. Security and Control, Cost Control, Cloud Sprawl, Visibility, Skills…..

Best practices to help get out of a messy middle, but Sharing the Pain: Core challenges for DevOps and DBAs (Operational challenges, evolving DevOps and business pressures, Heterogeneity, and Data Governance).

Six Core Data Desires – Data Mobility, Data Integrity and Quality, Data Availability, Cyber and Ransomware Resilience, Data Integration, Secure Data Access from anywhere.

One of the big discussion points is breaking down silos, automation, and making things go.

Building automation on deployments and supplying blueprints for specific configurations that will help provision Infrastructure Provisioning including databases, application servers, cloud services, and web or file services……brings forth ease of use, independence, efficiency, compliant by design, and securely by design. – APG

“Increased automation does not sidestep controls”

Data needs to become part of the deployment process where applicable

Be efficient, be innovative, and be secure.

74% of IT teams are now using more than one data platform.

25% are using more than four data platforms.

18% are making daily changes

50% increase in changes at short notice between 2022-24

84% who utilize AI say it delivers improved productivity to reduce time spent on DB deployments, Amplify the signal in the noise, and accelerate time-to-market

68% don’t collaborate between developers and operations. Bridging the gap between development and database operations.

How Redgate Flyway Can Boost Your DevOps Journey

Leave a reply

A brief introduction to the tool and its advantages for database migrations


UntitledDevOps is a culture and a set of practices that aim to deliver software faster and more reliably, by breaking down the silos between development and operations teams. One of the key aspects of DevOps is continuous integration and continuous delivery (CI/CD), which automates the building, testing, and deploying of code changes. However, CI/CD often overlooks a crucial component of software development: the database.

Database migrations, or the changes to the database schema and data, are often done manually, or with scripts that are not version-controlled, tested, or integrated with the rest of the codebase. This can lead to errors, delays, and inconsistencies, as well as security and compliance risks. Moreover, database migrations can be complex and risky, especially when dealing with large or legacy databases, or when working with multiple environments and teams.

That’s where Redgate Flyway comes in. Flyway is a tool that simplifies and automates database migrations, by allowing you to define them as code, version them, and apply them across different environments. Flyway supports a wide range of databases, including SQL Server, Oracle, MySQL, PostgreSQL, and more. Flyway also integrates with popular CI/CD tools, such as Jenkins, Azure DevOps, GitHub Actions, and more.

Flyway comes in different editions: Community, Teams, and Enterprise. The capabilities vary between editions, with the Enterprise edition offering auto-generation of migration and undo scripts, drift detection, change reporting, and more.

How does Flyway work?

Flyway works by using migration scripts, which are SQL files that contain the commands to alter the database schema or data. You can write these scripts yourself or use Flyway’s baseline feature to generate them from an existing database. You can also use Flyway’s undo feature to revert a migration if something goes wrong.

Flyway keeps track of the migrations that have been applied to each database, by using a metadata table called flyway_schema_history. This table records the version, description, checksum, and status of each migration. Flyway uses this information to determine which migrations need to be applied, and in what order, when you run the flyway migrate command.

Flyway also supports repeatable migrations, which are scripts that run every time you migrate, and always in the same order. These are useful for things like views, functions, or stored procedures, that depend on the database state. Flyway also supports callbacks, which are scripts that run before or after a migration, or on certain events, such as errors or warnings. These are useful for things like logging, auditing, or notifications.

What are the benefits of using Flyway?

Using Flyway can provide several benefits for your DevOps initiatives, such as:

  • Improved quality and reliability: By defining your database migrations as code, you can apply the same standards and practices that you use for your application code, such as version control, testing, code review, and documentation. This can help you avoid errors, inconsistencies, and technical debt, and ensure that your database is always in sync with your application. Flyway Enterprise can also auto-generate migration and undo scripts for SQL Server, Postgres, Oracle, and MySQL. This removes an element of human error while also introducing standardization of scripts and higher team productivity into the deployment process.
  • Faster and safer deployments: By automating your database migrations, you can reduce the time and effort required to deploy them and eliminate the need for manual interventions or downtime. You can also integrate Flyway with your CI/CD pipeline and run your migrations as part of your build or release process. This can help you achieve faster feedback loops and deliver value to your customers more frequently and reliably.
  • Increased collaboration and visibility: By using Flyway, you can enable better communication and collaboration between your development and operations teams, as well as other stakeholders, such as database administrators, testers, or auditors. You can also use Flyway’s features, such as the info, validate, or history commands, to get more visibility and insight into the state and history of your database migrations.
  • Enhanced security and compliance: By using Flyway, you can ensure that your database migrations are done in a consistent and controlled manner and that they adhere to the security and compliance requirements of your organization. You can also use Flyway’s features, such as the repair, clean, or undo commands, to fix any issues or restore your database to a previous state, in case of a problem or an audit.

What are some use cases and ROI savings for using Flyway?

Flyway can be used for a variety of use cases and scenarios, such as:
Migrating from one database platform to another or upgrading to a newer version of the same platform.

  • Refactoring or modernizing your database schema or data, to improve performance, scalability, or maintainability.
  • Managing multiple database environments, such as development, testing, staging, or production, and ensuring that they are aligned and consistent.
  • Supporting multiple teams or projects, that work on different parts of the same database, or on different databases, and need to coordinate and synchronize their changes.
  • Integrating with other tools or services, such as source control, testing, monitoring, or reporting, to enhance your DevOps workflow and capabilities.

Using Flyway can also provide significant ROI savings for your organization, by:

  • Reducing the cost and complexity of database migrations, eliminating the need for manual work, custom scripts, or third-party tools.
  • Increasing the productivity and efficiency of your teams, by enabling them to focus on delivering value, rather than dealing with database issues.
  • Minimizing the risk and impact of database failures, by ensuring that your migrations are tested, verified, and reversible.
  • Boosting the satisfaction and loyalty of your customers, by delivering high-quality software, faster and more frequently.

What is data classification, and why is it important?

1 Reply

DataClassificaiton
The benefits of data classification and the features of a tool like Microsoft Purview, a unified data governance service.

Data classification organizes data into categories based on its type, sensitivity, value, and usage. Data classification helps organizations at all levels to:

  • Protect sensitive and confidential data from unauthorized access, misuse, or loss.
  • Comply with data privacy and security regulations, such as GDPR, HIPAA, or CCPA.
  • Improve data quality, accuracy, and consistency to increase reliability; enhance data analysis, reporting, and decision-making by making the data more accessible and easily understood.
  • Comply with data privacy and security regulations, such as GDPR, HIPAA, or CCPA.
  • Optimize data storage, backup, and archiving strategies.
  • Improve data quality, accuracy, and consistency.
  • Enhance data analysis, reporting, and decision-making.

Data classification is not a one-time activity but a continuous process requiring regular monitoring and updating. However, data classification can be challenging, especially for large and complex data environments. Some of the common challenges I’ve ran into in the past are:

  • Lack of visibility and control over the data sources, locations, and flows.
  • Inconsistent or missing data labels, metadata, and tags.
  • Manual and time-consuming data classification processes.
  • Difficulty in enforcing data policies and standards across the organization.
  • High costs and risks of data breaches, fines, or reputational damage.

Data classification is also essential for dealing with large volumes of sensitive and regulated data, such as customer information, transaction records, credit scores, and financial statements. Data classification can help enterprise estates to:

  • Prevent data leaks, fraud, or identity theft that can harm customers and the institution’s reputation.
  • Meet the compliance requirements of various regulators, such as the Financial Conduct Authority (FCA), the Securities and Exchange Commission (SEC), or the Federal Reserve.
  • Reduce data storage and management costs by identifying and deleting redundant, obsolete, or trivial data.
  • Improve the data quality and reliability by detecting and correcting errors, inconsistencies, or anomalies.
  • Provide relevant and accurate data to enhance data analysis and reporting capabilities, supporting business intelligence, risk management, and customer service.

How can Microsoft Purview help with data classification?

Microsoft Purview is a unified data governance service that can help organizations discover, catalog, classify, and manage their data assets across on-premises, cloud, and hybrid environments. Microsoft Purview enables organizations to:

  • Automatically scan and catalog data sources, such as SQL Server, Azure Data Lake Storage, Azure Synapse Analytics, Power BI, and more.
  • Apply built-in or custom data classifications to identify and label sensitive or business-critical data.
  • Use a data map to visualize the data lineage, relationships, and dependencies.
  • Search and browse the data catalog using natural language queries or filters.
  • Access data insights and metrics, such as data quality, freshness, popularity, and compliance status.
  • Define and enforce data policies and standards across the organization.
  • Integrate with Azure Purview Data Catalog, Azure Synapse Analytics, Azure Data Factory, and other Azure services to enable end-to-end data governance and analytics.

Data classification is a vital component of data governance and management. It helps organizations protect, optimize, and leverage their data assets. Tools like Microsoft Purview is a comprehensive data governance service that simplifies and automates data classification and other data governance tasks. With Microsoft Purview, organizations can gain more visibility, control, and value from their data.