first commit
This commit is contained in:
Robert Nasarek
commit
d5c8076e2f
3 changed files with 471 additions and 0 deletions
122
WISSKI-UPDATE-TESTING.md
Normal file
122
WISSKI-UPDATE-TESTING.md
Normal file
|
|
@ -0,0 +1,122 @@
|
||||||
|
# WissKI Table Update Data Integrity Testing
|
||||||
|
|
||||||
|
This directory contains scripts to test data integrity when updating WissKI tables. The scripts ensure that no data is lost or truncated during schema updates.
|
||||||
|
|
||||||
|
## Files
|
||||||
|
|
||||||
|
- `test-wisski-data-integrity.php` - PHP script to export and verify data
|
||||||
|
- `test-wisski-update.sh` - Bash wrapper script for the complete testing process
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
### Option 1: Automated Testing (Recommended)
|
||||||
|
|
||||||
|
Run the complete test process (export → update → verify):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./test-wisski-update.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
This script will:
|
||||||
|
1. Export all WissKI table data (row counts, checksums, sample rows)
|
||||||
|
2. Prompt you to run Drupal updates
|
||||||
|
3. Verify data integrity after updates
|
||||||
|
4. Generate a detailed report
|
||||||
|
|
||||||
|
### Option 2: Manual Step-by-Step
|
||||||
|
|
||||||
|
#### Step 1: Export Data Before Update
|
||||||
|
|
||||||
|
```bash
|
||||||
|
php test-wisski-data-integrity.php export
|
||||||
|
```
|
||||||
|
|
||||||
|
This creates a directory `wisski-data-integrity-test/` with:
|
||||||
|
- `before-update-summary.json` - Summary of all tables
|
||||||
|
- Individual table JSON files with row counts, checksums, and sample data
|
||||||
|
|
||||||
|
#### Step 2: Run Drupal Updates
|
||||||
|
|
||||||
|
```bash
|
||||||
|
drush updatedb -y
|
||||||
|
```
|
||||||
|
|
||||||
|
Or use the Drupal admin interface: `/update.php`
|
||||||
|
|
||||||
|
#### Step 3: Verify Data Integrity
|
||||||
|
|
||||||
|
```bash
|
||||||
|
php test-wisski-data-integrity.php verify
|
||||||
|
```
|
||||||
|
|
||||||
|
This compares the current state with the exported data and reports:
|
||||||
|
- Row count changes
|
||||||
|
- Checksum mismatches (data changes)
|
||||||
|
- Sample data differences
|
||||||
|
|
||||||
|
## Output
|
||||||
|
|
||||||
|
All test results are saved in `wisski-data-integrity-test/`:
|
||||||
|
|
||||||
|
- `before-update-summary.json` - Complete export before update
|
||||||
|
- `verification-report.json` - Detailed verification results
|
||||||
|
- Individual table JSON files for detailed inspection
|
||||||
|
|
||||||
|
## What Gets Tested
|
||||||
|
|
||||||
|
For each WissKI table (all tables with `wisski` prefix):
|
||||||
|
|
||||||
|
1. **Row Count** - Ensures no rows are lost or added unexpectedly
|
||||||
|
2. **Data Checksum** - MD5 hash of all table data to detect any changes
|
||||||
|
3. **Sample Rows** - First 10 rows compared to detect data corruption
|
||||||
|
4. **Column Information** - Schema changes are logged
|
||||||
|
|
||||||
|
## Interpreting Results
|
||||||
|
|
||||||
|
### ✓ Success
|
||||||
|
- All row counts match
|
||||||
|
- All checksums match
|
||||||
|
- No data loss detected
|
||||||
|
|
||||||
|
### ⚠️ Issues Detected
|
||||||
|
- **Row count mismatch**: Rows were added or deleted
|
||||||
|
- **Checksum mismatch**: Data content changed (may be expected for schema updates)
|
||||||
|
- **Sample data mismatch**: First rows differ (investigate for corruption)
|
||||||
|
|
||||||
|
## Important Notes
|
||||||
|
|
||||||
|
1. **Backup First**: Always backup your database before running updates
|
||||||
|
2. **Schema Changes**: Some checksum changes may be expected if column types change (e.g., VARCHAR to TEXT)
|
||||||
|
3. **Large Tables**: For very large tables, checksum calculation may fall back to partial checksums
|
||||||
|
4. **Empty Tables**: Empty tables are marked with checksum 'empty' and skipped in checksum comparison
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
If verification fails:
|
||||||
|
|
||||||
|
1. Check `verification-report.json` for detailed information
|
||||||
|
2. Compare individual table JSON files before/after
|
||||||
|
3. Review Drupal update logs for errors
|
||||||
|
4. Check if schema changes are expected (column type changes, etc.)
|
||||||
|
|
||||||
|
## Example Output
|
||||||
|
|
||||||
|
```
|
||||||
|
Exporting data for 450 WissKI tables...
|
||||||
|
Processing wisski_entity_map...
|
||||||
|
Processing wisski_calling_bundles...
|
||||||
|
...
|
||||||
|
|
||||||
|
✓ Data export complete.
|
||||||
|
|
||||||
|
Verifying data integrity for 450 tables...
|
||||||
|
Checking wisski_entity_map...
|
||||||
|
✓ Row count matches: 1234
|
||||||
|
✓ Checksum matches
|
||||||
|
✓ Sample data matches
|
||||||
|
|
||||||
|
==========================================
|
||||||
|
✓ SUCCESS: All data verified successfully!
|
||||||
|
==========================================
|
||||||
|
```
|
||||||
|
|
||||||
274
test-wisski-data-integrity.php
Normal file
274
test-wisski-data-integrity.php
Normal file
|
|
@ -0,0 +1,274 @@
|
||||||
|
<?php
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @file
|
||||||
|
* Script to test WissKI table data integrity before and after schema updates.
|
||||||
|
*
|
||||||
|
* Usage:
|
||||||
|
* php test-wisski-data-integrity.php export # Export data before update
|
||||||
|
* php test-wisski-data-integrity.php verify # Verify data after update
|
||||||
|
*/
|
||||||
|
|
||||||
|
// Bootstrap Drupal.
|
||||||
|
$autoloader = require_once __DIR__ . '/autoload.php';
|
||||||
|
$request = \Symfony\Component\HttpFoundation\Request::createFromGlobals();
|
||||||
|
$kernel = \Drupal\Core\DrupalKernel::createFromRequest($request, $autoloader, 'prod');
|
||||||
|
$kernel->boot();
|
||||||
|
|
||||||
|
// Database connection.
|
||||||
|
$connection = \Drupal::database();
|
||||||
|
$outputDir = __DIR__ . '/wisski-data-integrity-test';
|
||||||
|
|
||||||
|
// Get all WissKI tables.
|
||||||
|
function getWisskiTables($connection) {
|
||||||
|
$tables = [];
|
||||||
|
// Query information_schema to get all WissKI tables.
|
||||||
|
$database = $connection->getConnectionOptions();
|
||||||
|
$dbName = $database['database'];
|
||||||
|
$result = $connection->query("SELECT TABLE_NAME FROM information_schema.TABLES WHERE TABLE_SCHEMA = :db AND TABLE_NAME LIKE 'wisski%'", [
|
||||||
|
':db' => $dbName,
|
||||||
|
]);
|
||||||
|
while ($row = $result->fetchAssoc()) {
|
||||||
|
$tables[] = $row['TABLE_NAME'];
|
||||||
|
}
|
||||||
|
return $tables;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Export table data: row count, checksum, sample rows.
|
||||||
|
function exportTableData($connection, $tableName, $outputDir) {
|
||||||
|
$data = [
|
||||||
|
'table' => $tableName,
|
||||||
|
'timestamp' => date('Y-m-d H:i:s'),
|
||||||
|
'row_count' => 0,
|
||||||
|
'checksum' => '',
|
||||||
|
'sample_rows' => [],
|
||||||
|
'column_info' => [],
|
||||||
|
];
|
||||||
|
|
||||||
|
// Get row count.
|
||||||
|
$countResult = $connection->query("SELECT COUNT(*) as cnt FROM {" . $tableName . "}");
|
||||||
|
$data['row_count'] = (int) $countResult->fetchField();
|
||||||
|
|
||||||
|
if ($data['row_count'] === 0) {
|
||||||
|
$data['checksum'] = 'empty';
|
||||||
|
return $data;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Get column information.
|
||||||
|
$columnsResult = $connection->query("SHOW COLUMNS FROM {" . $tableName . "}");
|
||||||
|
$columns = [];
|
||||||
|
while ($col = $columnsResult->fetchAssoc()) {
|
||||||
|
$columns[] = $col['Field'];
|
||||||
|
}
|
||||||
|
$data['column_info'] = $columns;
|
||||||
|
|
||||||
|
// Get sample rows (first 10 rows).
|
||||||
|
$sampleResult = $connection->query("SELECT * FROM {" . $tableName . "} LIMIT 10");
|
||||||
|
$samples = [];
|
||||||
|
while ($row = $sampleResult->fetchAssoc()) {
|
||||||
|
$samples[] = $row;
|
||||||
|
}
|
||||||
|
$data['sample_rows'] = $samples;
|
||||||
|
|
||||||
|
// Calculate checksum of all data.
|
||||||
|
// For large tables, we'll use a hash of all row data concatenated.
|
||||||
|
// Build checksum query - use backticks for column names.
|
||||||
|
$checksumFields = [];
|
||||||
|
foreach ($columns as $col) {
|
||||||
|
$checksumFields[] = "COALESCE(CAST(`" . $col . "` AS CHAR), '')";
|
||||||
|
}
|
||||||
|
$orderByCols = array_slice($columns, 0, min(5, count($columns)));
|
||||||
|
$orderBy = implode(', ', array_map(function($col) {
|
||||||
|
return "`" . $col . "`";
|
||||||
|
}, $orderByCols));
|
||||||
|
|
||||||
|
// Use CONCAT_WS for safer concatenation.
|
||||||
|
$concatExpr = "CONCAT_WS('|', " . implode(', ', $checksumFields) . ")";
|
||||||
|
$checksumQuery = "SELECT MD5(GROUP_CONCAT(" . $concatExpr .
|
||||||
|
" ORDER BY " . $orderBy . " SEPARATOR '|||')) as chksum FROM {" . $tableName . "}";
|
||||||
|
|
||||||
|
try {
|
||||||
|
$checksumResult = $connection->query($checksumQuery);
|
||||||
|
$data['checksum'] = $checksumResult->fetchField() ?: 'no_checksum';
|
||||||
|
}
|
||||||
|
catch (\Exception $e) {
|
||||||
|
// If checksum fails (e.g., GROUP_CONCAT too large), use row count + first row hash.
|
||||||
|
$data['checksum'] = 'partial_' . md5($data['row_count'] . serialize($samples[0] ?? []));
|
||||||
|
}
|
||||||
|
|
||||||
|
return $data;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Export all WissKI tables.
|
||||||
|
function exportAllTables($connection, $outputDir) {
|
||||||
|
if (!is_dir($outputDir)) {
|
||||||
|
mkdir($outputDir, 0755, TRUE);
|
||||||
|
}
|
||||||
|
|
||||||
|
$tables = getWisskiTables($connection);
|
||||||
|
$allData = [
|
||||||
|
'export_timestamp' => date('Y-m-d H:i:s'),
|
||||||
|
'tables' => [],
|
||||||
|
];
|
||||||
|
|
||||||
|
echo "Exporting data for " . count($tables) . " WissKI tables...\n";
|
||||||
|
|
||||||
|
foreach ($tables as $table) {
|
||||||
|
echo " Processing {$table}...\n";
|
||||||
|
$tableData = exportTableData($connection, $table, $outputDir);
|
||||||
|
$allData['tables'][$table] = $tableData;
|
||||||
|
|
||||||
|
// Save individual table export.
|
||||||
|
$tableFile = $outputDir . '/' . $table . '.json';
|
||||||
|
file_put_contents($tableFile, json_encode($tableData, JSON_PRETTY_PRINT));
|
||||||
|
}
|
||||||
|
|
||||||
|
// Save summary.
|
||||||
|
$summaryFile = $outputDir . '/before-update-summary.json';
|
||||||
|
file_put_contents($summaryFile, json_encode($allData, JSON_PRETTY_PRINT));
|
||||||
|
|
||||||
|
echo "\nExport complete! Data saved to: {$outputDir}\n";
|
||||||
|
echo "Summary: {$summaryFile}\n";
|
||||||
|
|
||||||
|
return $allData;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Verify data integrity after update.
|
||||||
|
function verifyDataIntegrity($connection, $outputDir) {
|
||||||
|
$summaryFile = $outputDir . '/before-update-summary.json';
|
||||||
|
|
||||||
|
if (!file_exists($summaryFile)) {
|
||||||
|
echo "ERROR: Before-update summary not found at: {$summaryFile}\n";
|
||||||
|
echo "Please run 'export' first.\n";
|
||||||
|
return FALSE;
|
||||||
|
}
|
||||||
|
|
||||||
|
$beforeData = json_decode(file_get_contents($summaryFile), TRUE);
|
||||||
|
$issues = [];
|
||||||
|
$verified = [];
|
||||||
|
|
||||||
|
echo "Verifying data integrity for " . count($beforeData['tables']) . " tables...\n\n";
|
||||||
|
|
||||||
|
foreach ($beforeData['tables'] as $tableName => $beforeTableData) {
|
||||||
|
echo "Checking {$tableName}...\n";
|
||||||
|
|
||||||
|
$afterTableData = exportTableData($connection, $tableName, $outputDir);
|
||||||
|
|
||||||
|
// Check row count.
|
||||||
|
if ($afterTableData['row_count'] !== $beforeTableData['row_count']) {
|
||||||
|
$issues[] = [
|
||||||
|
'table' => $tableName,
|
||||||
|
'issue' => 'row_count_mismatch',
|
||||||
|
'before' => $beforeTableData['row_count'],
|
||||||
|
'after' => $afterTableData['row_count'],
|
||||||
|
'difference' => $afterTableData['row_count'] - $beforeTableData['row_count'],
|
||||||
|
];
|
||||||
|
echo " ⚠️ ROW COUNT MISMATCH: Before={$beforeTableData['row_count']}, After={$afterTableData['row_count']}\n";
|
||||||
|
}
|
||||||
|
else {
|
||||||
|
echo " ✓ Row count matches: {$afterTableData['row_count']}\n";
|
||||||
|
}
|
||||||
|
|
||||||
|
// Check checksum.
|
||||||
|
if ($beforeTableData['checksum'] !== 'empty' && $afterTableData['checksum'] !== 'empty') {
|
||||||
|
if ($beforeTableData['checksum'] !== $afterTableData['checksum']) {
|
||||||
|
$issues[] = [
|
||||||
|
'table' => $tableName,
|
||||||
|
'issue' => 'checksum_mismatch',
|
||||||
|
'before' => $beforeTableData['checksum'],
|
||||||
|
'after' => $afterTableData['checksum'],
|
||||||
|
];
|
||||||
|
echo " ⚠️ CHECKSUM MISMATCH: Data may have changed!\n";
|
||||||
|
}
|
||||||
|
else {
|
||||||
|
echo " ✓ Checksum matches\n";
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Compare sample rows if available.
|
||||||
|
if (!empty($beforeTableData['sample_rows']) && !empty($afterTableData['sample_rows'])) {
|
||||||
|
$sampleMatch = TRUE;
|
||||||
|
$minSamples = min(count($beforeTableData['sample_rows']), count($afterTableData['sample_rows']));
|
||||||
|
for ($i = 0; $i < $minSamples; $i++) {
|
||||||
|
if ($beforeTableData['sample_rows'][$i] !== $afterTableData['sample_rows'][$i]) {
|
||||||
|
$sampleMatch = FALSE;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (!$sampleMatch) {
|
||||||
|
$issues[] = [
|
||||||
|
'table' => $tableName,
|
||||||
|
'issue' => 'sample_data_mismatch',
|
||||||
|
];
|
||||||
|
echo " ⚠️ Sample data differs!\n";
|
||||||
|
}
|
||||||
|
else {
|
||||||
|
echo " ✓ Sample data matches\n";
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (empty(array_filter($issues, function($issue) use ($tableName) {
|
||||||
|
return $issue['table'] === $tableName;
|
||||||
|
}))) {
|
||||||
|
$verified[] = $tableName;
|
||||||
|
}
|
||||||
|
|
||||||
|
echo "\n";
|
||||||
|
}
|
||||||
|
|
||||||
|
// Save verification report.
|
||||||
|
$report = [
|
||||||
|
'verification_timestamp' => date('Y-m-d H:i:s'),
|
||||||
|
'tables_verified' => count($verified),
|
||||||
|
'tables_with_issues' => count($issues),
|
||||||
|
'verified_tables' => $verified,
|
||||||
|
'issues' => $issues,
|
||||||
|
];
|
||||||
|
|
||||||
|
$reportFile = $outputDir . '/verification-report.json';
|
||||||
|
file_put_contents($reportFile, json_encode($report, JSON_PRETTY_PRINT));
|
||||||
|
|
||||||
|
// Print summary.
|
||||||
|
echo "\n" . str_repeat('=', 60) . "\n";
|
||||||
|
echo "VERIFICATION SUMMARY\n";
|
||||||
|
echo str_repeat('=', 60) . "\n";
|
||||||
|
echo "Tables verified: " . count($verified) . "\n";
|
||||||
|
echo "Tables with issues: " . count($issues) . "\n";
|
||||||
|
|
||||||
|
if (!empty($issues)) {
|
||||||
|
echo "\n⚠️ ISSUES FOUND:\n";
|
||||||
|
foreach ($issues as $issue) {
|
||||||
|
echo " - {$issue['table']}: {$issue['issue']}\n";
|
||||||
|
if (isset($issue['difference'])) {
|
||||||
|
echo " Difference: {$issue['difference']} rows\n";
|
||||||
|
}
|
||||||
|
}
|
||||||
|
echo "\nFull report: {$reportFile}\n";
|
||||||
|
return FALSE;
|
||||||
|
}
|
||||||
|
else {
|
||||||
|
echo "\n✓ All tables verified successfully! No data loss detected.\n";
|
||||||
|
echo "Report: {$reportFile}\n";
|
||||||
|
return TRUE;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Main execution.
|
||||||
|
$command = $argv[1] ?? 'help';
|
||||||
|
|
||||||
|
switch ($command) {
|
||||||
|
case 'export':
|
||||||
|
exportAllTables($connection, $outputDir);
|
||||||
|
break;
|
||||||
|
|
||||||
|
case 'verify':
|
||||||
|
verifyDataIntegrity($connection, $outputDir);
|
||||||
|
break;
|
||||||
|
|
||||||
|
default:
|
||||||
|
echo "Usage:\n";
|
||||||
|
echo " php test-wisski-data-integrity.php export # Export data before update\n";
|
||||||
|
echo " php test-wisski-data-integrity.php verify # Verify data after update\n";
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
|
||||||
75
test-wisski-update.sh
Executable file
75
test-wisski-update.sh
Executable file
|
|
@ -0,0 +1,75 @@
|
||||||
|
#!/bin/bash
|
||||||
|
|
||||||
|
# Script to test WissKI table updates with data integrity checks.
|
||||||
|
# This script exports data before update, runs the update, and verifies data integrity.
|
||||||
|
|
||||||
|
set -e
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
cd "$SCRIPT_DIR"
|
||||||
|
|
||||||
|
OUTPUT_DIR="$SCRIPT_DIR/wisski-data-integrity-test"
|
||||||
|
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
|
||||||
|
|
||||||
|
echo "=========================================="
|
||||||
|
echo "WissKI Table Update Data Integrity Test"
|
||||||
|
echo "=========================================="
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Step 1: Export data before update.
|
||||||
|
echo "Step 1: Exporting data BEFORE update..."
|
||||||
|
echo "----------------------------------------"
|
||||||
|
php test-wisski-data-integrity.php export
|
||||||
|
|
||||||
|
if [ ! -f "$OUTPUT_DIR/before-update-summary.json" ]; then
|
||||||
|
echo "ERROR: Failed to export data before update!"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "✓ Data export complete. Backup saved to: $OUTPUT_DIR"
|
||||||
|
echo ""
|
||||||
|
read -p "Press Enter to continue with the update, or Ctrl+C to cancel..."
|
||||||
|
|
||||||
|
# Step 2: Run Drupal updates.
|
||||||
|
echo ""
|
||||||
|
echo "Step 2: Running Drupal updates..."
|
||||||
|
echo "----------------------------------------"
|
||||||
|
|
||||||
|
# Check if drush is available.
|
||||||
|
if command -v drush &> /dev/null; then
|
||||||
|
echo "Running: drush updatedb -y"
|
||||||
|
drush updatedb -y
|
||||||
|
else
|
||||||
|
echo "Drush not found. Please run Drupal updates manually:"
|
||||||
|
echo " drush updatedb -y"
|
||||||
|
echo ""
|
||||||
|
read -p "Press Enter after you have run the updates, or Ctrl+C to cancel..."
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Step 3: Verify data integrity.
|
||||||
|
echo ""
|
||||||
|
echo "Step 3: Verifying data integrity AFTER update..."
|
||||||
|
echo "----------------------------------------"
|
||||||
|
php test-wisski-data-integrity.php verify
|
||||||
|
|
||||||
|
VERIFICATION_EXIT=$?
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
if [ $VERIFICATION_EXIT -eq 0 ]; then
|
||||||
|
echo "=========================================="
|
||||||
|
echo "✓ SUCCESS: All data verified successfully!"
|
||||||
|
echo "=========================================="
|
||||||
|
else
|
||||||
|
echo "=========================================="
|
||||||
|
echo "⚠️ WARNING: Issues detected during verification!"
|
||||||
|
echo "Check the verification report for details."
|
||||||
|
echo "=========================================="
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "Test results saved to: $OUTPUT_DIR"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
exit $VERIFICATION_EXIT
|
||||||
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue