Read Me - Orchestrate.procfwk

For complete documentation on this solution see procfwk.com.

ProcFwk Has Become CF.Cumulus.Control

See blog: mrpaulandrew.com

See new product page: cloudformations.org/cumulus

ProcFwk will receive no further development beyond December 2023.

Framework Capabilities

• Granular metadata control.
• Metadata integrity checking.
• Global properties.
• Complete pipeline dependency chains.
• Concurrent batch executions (hourly/daily/monthly).
• Execution restart-ability.
• Parallel pipeline execution.
• Full execution and error logs.
• Operational dashboards.
• Low cost orchestration.
• Disconnection between framework and Worker pipelines.
• Cross Tenant/Subscription/Data Factory control flows.
• Pipeline parameter support.
• Simple troubleshooting.
• Easy deployment.
• Email alerting.
• Automated testing.
• Azure Key Vault integration.
• Is pipeline already running checks.

Complete Data Factory Activity Chain

Issues

If you've found a bug or have a new feature request please log the details using the repository issues.

Go to... Issues

Projects

Go to... External Requests

Go to... Internal Backlog

Release Details

| Version | Overview | Version Details & Release Notes | |:----:|--------------|--------| | 2.0 |Azure Synapse Analytics fully supported as an interchangeable orchestrator of pipelines within the procfwk.|GitHub Pages: </br>Orchestrators</br>Orchestrator Types </br></br>Release Summary Video: </br>YouTube - procfwk Playlist<br/><br/>GitHub Issues:</br>procfwk #95 | | 2.0-beta |<u>Azure Synapse Analytics Beta support</u> added.<br/><br/>Development of Azure Functions App completed using the Synapse namespace: Azure.Analytics.Synapse.Artifacts with version 1.0.0-beta.1 of the NuGet package.|GitHub Issues:</br>procfwk #21 | | 1.9.2 |<u>Batch Executions</u> added, plus:<ul><li>Exception Pipeline</li><li>Running Pipeline Check</li><li>Pipeline Parameter Last Values</li><li>Worker Pipeline Validation</li></ul>|GitHub Pages: Batch Executions </br></br>Release Demo Summary Video: YouTube - procfwk Playlist<br/><br/>GitHub Issues:</br>procfwk #78<br/>procfwk #77<br/>procfwk #71<br/>procfwk #73<br/>procfwk #80<br/>procfwk #72 | | 1.9.1 |<u>Activity Policy Update</u>, plus:<ul><li>Secure Activity Inputs/Outputs.</li><li>Execution Wrapper Hardening.</li><li>New Activity Icons and Framework Factory Cosmetics.</li></ul> |GitHub Issues:</br>procfwk #65<br/>procfwk #66<br/>procfwk #67<br/>procfwk #69 | | 1.9.0 |<u>Cross Tenant & Subscription Support</u> added, plus:<ul><li>New integration tests created.</li><li>Infant pipeline refactoring.</li><li>tSQLt project added.</li></ul> |GitHub Issues:</br>procfwk #34<br/>procfwk #35<br/>procfwk #46<br/>procfwk #55<br/>procfwk #56<br/>procfwk #59 | | 1.8.6 |<u>Pipeline Expressions Refactored to Use Variables</u> added, plus:<ul><li>New integration tests created.</li><li>Complete activity chain redrawn in Visio.</li></ul> |GitHub Issues:</br>procfwk #51<br/>procfwk #52 | | 1.8.5 |<u>Execution Precursor</u> added, plus:<ul><li>PowerShell helper to add initial Worker metadata.</li></ul> |procfwk v1.8.5 - Execution Precursor | | 1.8.4 |<u>Database Schema Reorganise and Restructuring</u> |procfwk v1.8.4 - Database Schema Reorganise and Restructuring | | 1.8.3 |<u>Bug Fixes from the Community</u>, including:<ul><li>Email alerts sent to blank email addresses due to wrong flow in Child pipeline.</li><li>Worker pipelines cancelled during an execution fail when the framework is restarted due to missing Parent pipeline clean up condition.</li></ul> |GitHub Issues:</br>procfwk #38<br/>procfwk #37 | | 1.8.2 |<u>Optionally Store SPN Details in Azure Key Vault</u> |procfwk v1.8.2 - Optionally Store SPN Details in Azure Key Vault | | 1.8.1 |<u>Automated Framework Pipeline Testing</u> added, including tests for:<ul><li>A simple grandparent run.</li><li>All types of failure dependency handling.</li><li>Metadata checks when pipelines and staged are disabled.</li><li>No pipeline parameters provided.</li></ul> |Blog Series: <br/><ol><li>Set up automated testing for Azure Data Factory</li><li>Automate integration tests in Azure Data Factory</li><li>Isolated functional tests for Azure Data Factory</li><li>Testing Azure Data Factory in your CI/CD pipeline</li><li>Unit testing Azure Data Factory pipelines</li><li>Calculating Azure Data Factory test coverage</li></ol> | | 1.8.0 |<u>Complete Pipeline Dependency Chains For Failure Handling</u> added, plus:<ul><li>Clean up of a previous execution run if Workers appear as running.</li><li>New metadata integrity checks.</li><li>Internal get property value function added.</li></ul> |procfwk v1.8 - Complete Pipeline Dependency Chains For Failure Handling | | 1.7.3 |Data Factory Deployment Updated To Use azure.datafactory.tools PowerShell Module |SQLPlayer/azure.datafactory.tools | | 1.7.2 |<u>Pipeline Parameter NULL Handling</u> added, plus:<ul><li>Worker pipelines with a status of 'Running' protected from a new execution start/restart.</li></ul> |procfwk v1.7.2 - NULL Pipeline Parameters Handled | | 1.7.1 |<u>Alerting Check Bug Fix</u> added, plus:<ul><li>Pipeline parameter value size limit removed.</li></ul> |procfwk v1.7.1 - Alerting Bug Fix And Pipeline Parameter Size Limit Removed | | 1.7.0 |<u>Pipleline EMail Alerting</u> added, plus:<ul><li>Send email Function implemented and hardened.</li><li>Handy Notebook updates.</li><li>Activity failure paths improved.</li><li>MIT license and code of conduct added.</li><li>Error table bug fix. Error code attribute; INT to VARCHAR</li></ul> |procfwk v1.7 - Pipeline Email Alerting | | 1.6.0 |<u>Error Details for Failed Activities Captured</u>, plus:<ul><li>Pipeline parameters used at runtime captured in execution logs.</li><li>Emailing Function added, not yet implemented.</li><li>Unknown Worker outcomes optionally blocks downstream stages.</li><li>Solution housekeeping.</li></ul> |procfwk v1.6 - Error Details for Failed Activities Captured | | 1.5.0 |<u>Power BI Dashboard for Framework Executions</u>, plus:<ul><li>Worker Parallelism View.</li><li>Pipeline Run ID now logged.</li><li>Logging Attributes Bug Fix.</li></ul> |procfwk v1.5 - Power BI Dashboard for Framework Executions | | 1.4.0 |<u>Enhancements for Long Running Pipelines</u>, plus:<ul><li>Pipeline check status function added.</li><li>Function Data Factory client moved to internal class.</li><li>SQL G