ShareGate Data Storage Architecture - Interactive Research Study

🚀 Understanding Where and How ShareGate Stores Migration Data

Quick Links

• 🔐 ShareGate Login
• 📚 Documentation
• ⚡ PowerShell Reference
• 🌐 ShareGate Website

📁 Data Storage Locations

ShareGate stores its data in several key locations on your local machine. Understanding these locations is crucial for managing migration data effectively.

Primary Storage Paths

File Structure Overview

%LOCALAPPDATA%\Sharegate\userdata\
│
├── 📁 migration\
│   ├── RecentSessionsInformation.sdb  (100 KB - 1 MB)
│   ├── PrincipalMappings.sdb         (10 - 100 KB)
│   ├── CustomFields.sdb               (1 - 10 KB)
│   ├── NintexApiKey.sdb              (1 - 10 KB)
│   └── _Schema.sdb                    (Schema definitions)
│
├── 📁 applogs\                        ⚠️ LARGEST FILES = "Tasks" in UI
│   ├── 10000001.sdb                  (1 MB - 10+ GB per file!)
│   ├── 10000002.sdb                  (Each is a migration task/session)
│   ├── 10000003.sdb                  (Contains ALL item details)
│   └── ...                           (Numbers increment per session)
│
├── 📁 cache\                          ✅ SAFE TO DELETE
│   └── [Various temporary .sdb files]
│
├── 📁 scheduling\
│   └── [Scheduled job definitions]
│
└── 📁 tables\
    └── Reports.sdb                    (Report configurations)

🔧 PowerShell Commands

ShareGate provides powerful PowerShell cmdlets for managing and exporting migration data. Here are the most important commands:

Find ShareGate Data

# Get ShareGate data location
$shareGateData = "$env:LOCALAPPDATA\Sharegate\userdata"

# List all .sdb files with sizes
Get-ChildItem -Path $shareGateData -Filter "*.sdb" -Recurse | 
    Select-Object FullName, @{N='Size(MB)';E={[Math]::Round($_.Length/1MB,2)}}, LastWriteTime |
    Sort-Object 'Size(MB)' -Descending

Export All Migration Sessions

# Export each migration session to individual CSV files
Import-Module Sharegate

$sessions = Find-CopySessions
Write-Host "$($sessions.Count) sessions found" -ForegroundColor Green

mkdir "ShareGate Migration Logs" -ErrorAction SilentlyContinue
Push-Location "ShareGate Migration Logs"

for($o=0; $o -lt $sessions.count; $o++){
    $session = $sessions[$o]
    # Both .CSV and .XLSX are supported:
    Export-Report $session -NoItemVersions -DefaultColumns -Path ".\$($session.Id).csv"
}

Pop-Location

Performance-Based Cleanup (Recommended)

# Export logs when performance degrades, then delete old logs
# Check total size first
$appLogsPath = "$env:LOCALAPPDATA\Sharegate\userdata\applogs"
$totalGB = (Get-ChildItem $appLogsPath -Filter "*.sdb" | 
    Measure-Object Length -Sum).Sum / 1GB

if ($totalGB -gt 10) {
    Write-Host "WARNING: Applogs size is $([Math]::Round($totalGB, 2)) GB" -ForegroundColor Red
    Write-Host "Export your sessions with Export-Report, then delete old logs" -ForegroundColor Yellow
    
    # After exporting, delete large old files
    Get-ChildItem $appLogsPath -Filter "*.sdb" | 
        Where-Object { $_.Length -gt 1GB } |
        Sort-Object LastWriteTime |
        Select-Object -First 10 |
        Remove-Item -Force -WhatIf  # Remove -WhatIf to actually delete
}

📊 Performance Impact Analysis

Task Loading Performance in ShareGate UI

When you click on a task in ShareGate’s Task History, the application needs to load and deserialize the entire .sdb file for that session. Larger files take significantly longer to load:

📸 ShareGate Tasks Interface

The applogs files (10000xxx.sdb) appear as “Tasks” in the ShareGate desktop application. Each task represents a migration session with detailed results:

The ShareGate Tasks interface showing migration sessions - each entry corresponds to a .sdb file in the applogs folder

Tasks = Logs

UI Name for Applogs

10000xxx.sdb

Actual File Format

Success/Error

Result Tracking

Export Available

Via PowerShell

🔄 Data Flow Architecture

ShareGate follows a specific data flow pattern for storing and retrieving migration data:

Process Flow

1. ShareGate Desktop → Performs Migration
2. SnapDb Engine → Serializes Objects
3. .sdb Files → Binary Storage
4. PowerShell → Export Reports

💾 DLL Reference and Technical Details

DLL Locations

• Primary DLL Location: %LOCALAPPDATA%\Apps\ShareGate\
• Data Storage Location: %LOCALAPPDATA%\Sharegate\userdata\

Note: There are TWO distinct locations - Apps\ShareGate\ for executables and Sharegate\userdata\ for data.

Core ShareGate DLLs

While it is not possible to reuse ShareGate’s DLLs directly due to heavy obfuscation, I’m documenting them here for scientific and educational purposes. Understanding the architecture helps migration specialists troubleshoot issues and optimize their workflows.

⚙️ ShareGate Configuration Files

ShareGate stores various configuration settings in JSON format. These files control different aspects of the application:

Configuration File Details

{
  "theme": "light",
  "maxConcurrentOperations": 4,
  "autoUpdate": true,
  "telemetryEnabled": true,
  "windowState": {
    "width": 1200,
    "height": 800,
    "maximized": false
  }
}

{
  "defaultAuthMethod": "Modern",
  "connectionTimeout": 30000,
  "useProxy": false,
  "validateCertificates": true,
  "retryAttempts": 3,
  "sslProtocol": "Tls12"
}

{
  "defaultCopyOptions": {
    "preservePermissions": true,
    "preserveMetadata": true,
    "versionLimit": 10,
    "includeVersions": false,
    "preserveAuthors": true
  },
  "exportFormat": "xlsx",
  "notificationSound": true
}

{
  "recentSites": [
    {
      "url": "https://[tenant].sharepoint.com",
      "title": "Contoso Portal",
      "lastAccessed": "2024-01-15T10:30:00Z",
      "isFavorite": true,
      "authType": "Modern"
    }
  ],
  "maxRecentItems": 20
}

# ShareGate Health Check Script
function Test-ShareGateHealth {
    $data = "$env:LOCALAPPDATA\Sharegate\userdata"
    
    # Check total size
    $totalGB = (Get-ChildItem $data -Recurse | 
        Measure-Object Length -Sum).Sum / 1GB
    
    # Check old logs count
    $oldLogs = Get-ChildItem "$data\applogs" -Filter "*.sdb" |
        Where-Object { $_.LastWriteTime -lt (Get-Date).AddDays(-90) }
    
    # Display health status
    Write-Host "`n=== ShareGate Health Status ===" -ForegroundColor Cyan
    
    if ($totalGB -lt 5) {
        Write-Host "✓ Size OK: $([Math]::Round($totalGB,1)) GB" -ForegroundColor Green
    } elseif ($totalGB -lt 10) {
        Write-Host "⚠ Size Warning: $([Math]::Round($totalGB,1)) GB" -ForegroundColor Yellow
    } else {
        Write-Host "✗ Size Critical: $([Math]::Round($totalGB,1)) GB" -ForegroundColor Red
    }
    
    if ($oldLogs.Count -gt 0) {
        Write-Host "⚠ Found $($oldLogs.Count) logs older than 90 days" -ForegroundColor Yellow
    } else {
        Write-Host "✓ No old logs found" -ForegroundColor Green
    }
}

Test-ShareGateHealth

# Backup ShareGate Data
$timestamp = Get-Date -Format "yyyyMMdd-HHmmss"
$backupPath = "$env:USERPROFILE\Desktop\ShareGate-Backup-$timestamp"

# Create backup directory
New-Item -ItemType Directory -Path $backupPath -Force

# Backup migration data (excluding cache)
$source = "$env:LOCALAPPDATA\Sharegate\userdata"
$folders = @('migration', 'scheduling', 'tables')

foreach ($folder in $folders) {
    Copy-Item -Path "$source\$folder" -Destination $backupPath -Recurse
    Write-Host "✓ Backed up: $folder" -ForegroundColor Green
}

Write-Host "`n✓ Backup complete: $backupPath" -ForegroundColor Green

# Performance-Based Cleanup Script (Run ONLY when ShareGate is slow)
$shareGateData = "$env:LOCALAPPDATA\Sharegate\userdata"

# 1. Clear cache
Remove-Item "$shareGateData\cache\*" -Recurse -Force -ErrorAction SilentlyContinue
Write-Host "✓ Cache cleared" -ForegroundColor Green

# 2. Archive old applogs (>90 days)
$archivePath = "$env:USERPROFILE\Desktop\ShareGate-Archive-$(Get-Date -Format 'yyyyMM')"
New-Item -ItemType Directory -Path $archivePath -Force

$oldLogs = Get-ChildItem "$shareGateData\applogs" -Filter "*.sdb" |
    Where-Object { $_.LastWriteTime -lt (Get-Date).AddDays(-90) }

if ($oldLogs) {
    $oldLogs | Move-Item -Destination $archivePath
    Write-Host "✓ Archived $($oldLogs.Count) old log files" -ForegroundColor Green
}

# 3. Display space recovered
$newSize = (Get-ChildItem $shareGateData -Recurse | 
    Measure-Object Length -Sum).Sum / 1GB
Write-Host "✓ Current size: $([Math]::Round($newSize, 2)) GB" -ForegroundColor Green

⚡ Additional Technical Details

SnapDb Technology

• 📦 Binary key-value store
• 🌳 B+Tree indexing
• 🔐 Custom serialization
• ⚠️ No SQL support

Understanding .SDB Files (Serialization Details)

.SDB files are ShareGate’s proprietary storage format for Task History:

• What they are: Binary files containing serialized .NET objects
• Where created: In %LOCALAPPDATA%\Sharegate\userdata\applogs\ as numbered files (10000001.sdb, 10000002.sdb, etc.)
• When created: Each migration session creates a new .sdb file
• What they contain: Complete details of every item migrated, including success/failure status, metadata, and error messages
• How to inspect:
  • Via ShareGate UI: Click on any task in the Task History
  • Via PowerShell: Use Find-CopySessions and Export-Report cmdlets
• Why they get large: Each file contains ALL item details from that migration session - a migration of 100,000 items creates a massive .sdb file

PowerShell Cmdlets Reference

📊 Reporting Cmdlets

• Find-CopySessions - Read migration history
• Export-Report - Export to CSV/Excel

📁 Copy Cmdlets (Generate Logs)

• Copy-Content - Items between lists
• Copy-List - Entire lists
• Copy-Site - Site structure
• Copy-Team - Microsoft Teams
• Copy-PlannerPlan - Planner plans
• Import-Document - File system imports

📚 Full PowerShell Documentation →

Conclusion

This research provides a comprehensive understanding of ShareGate’s data storage architecture. The key findings reveal that ShareGate uses a custom NoSQL database (SnapDb) with serialized .NET objects rather than a traditional relational database. This architectural choice, while convenient for development, can lead to performance issues at scale.

Key Takeaways

1. Monitor your applogs folder size - Keep it under 10GB for optimal performance
2. Export data regularly - Use PowerShell cmdlets to export migration history
3. Clean up periodically - But only when performance degrades
4. Backup before cleanup - Always preserve your migration data
5. Understand the architecture - Knowledge helps in troubleshooting and optimization

About This Research

For SharePoint Migration Specialists seeking to understand ShareGate’s inner workings.