Automated Sitecore content synchronization using Item as Resources

Synchronizing content between Sitecore environments, especially copying production content to non-production environments, has always been challenging. The introductions of serialization tools, such as Unicorn, made it a little easier. With Sitecore 10, Sitecore released their very own Sitecore CLI, capable of pulling and pushing content from new APIs available in the CMS.

Additionally, Sitecore 10 came with a feature called “Items as Resources” (read more in Sitecore’s documentation). With this feature, you can create a .dat file which is read during the startup of Sitecore and inserts items stored in that file in the content tree, removing the requirement to always store all items in the Sitecore databases.

This feature comes with a plugin for Sitecore’s CLI. Using the commands in this plugin, you can create resource files based on your Sitecore Content Serialization (SCS) configuration and serialized items.

To create a resource file based on your SCS serialized items, you need to run the following command in PowerShell.

dotnet sitecore itemres create -o [file name]

This command will create a resource file based on all the SCS modules you have created. The serialized items you have on disk for those modules will be added to the file, and when deployed to Sitecore they will show up in the content tree.

The itemres command has two helpful options to limit the included SCS modules. Just like the ser command, it has the –include and –exclude options to either include or exclude SCS modules based on their name.
The following example shows how this can be used to exclude any modules in the Project layer:

dotnet sitecore itemres create -o [file name] -e Project.*

The file that is created will have to be deployed to both your CM and CD instances. By copying them to the ~/App_Data/items/ folder of you Sitecore instance, they will be read during the Sitecore startup process and included in the content editor.

Creating a resource file in your deployment process

We have now seen how you can use the itemres commands to create a resources file from your local repository, but when doing this for a content synchronization process there are some preliminary steps that need to be taken.

For a process to synchronize production content down to lower environments, you will want to create one or more SCS modules specific to the content that you want to synchronize.

Let’s take this SCS module as an example;

{
  "namespace": "Content.Corporate",
  "description": "Corporate Content",
  "items": {
    "includes": [
      {
        "database": "web",
        "name": "Corporate",
        "path": "/sitecore/content"
      },
      {
        "database": "web",
        "name": "Forms",
        "path": "/sitecore/Forms"
      }
    ]
  }
}

Note that in this example we configure the module to pull content from the web database and not master. This can be useful if you only want to synchronize content that has been published.

In order to create a resources file for this content in your deployment process, you will need to first pull this content from the environment you want to use as a source of your synchronization. This means you first need to login to the Sitecore environment using the Sitecore CLI, before being able to pull content.

Before you can run Sitecore CLI commands, you need to make sure the Azure DevOps build agent has all the tools required. The first steps we run as part of this synchronization pipeline will be to do a checkout of the repository, installing the right version of .NET for the CLI to run, and restoring the dotnet tools.

The following pipeline yaml snippet shows how you can run these tasks.

jobs:
- job: create_itemres_files
  displayName: 'Create Items as Resources files'
  steps:
  - checkout: self

  - task: UseDotNet@2
    displayName: Install .NET 6.x
    inputs:
      packageType: 'sdk'
      version: '6.x'

  - task: DotNetCoreCLI@2
    displayName: Dotnet Tool Restore
    inputs:
      command: 'custom'
      custom: 'tool'
      arguments: 'restore'

After executing these tasks, the build agent is ready to run Sitecore CLI commands. The next step is to login to Sitecore. For this, we need to make use of the feature to specify a client ID and secret during the login command.

  - task: DotNetCoreCLI@2
    displayName: Login to Sitecore CLI
    inputs:
      command: 'custom'
      custom: 'sitecore'
      arguments: 'login --cm ${{ parameters.sitecore_cm_url }} --auth ${{ parameters.sitecore_auth_url }} --allow-write true --client-credentials true --client-id CLI --client-secret ${{ parameters.sitecore_cli_secret }}'

Now that the build agent CLI is logged in to Sitecore, we can run the serialization pull command to pull the content from the Sitecore environment.

  - task: DotNetCoreCLI@2
    displayName: Sitecore ser pull
    inputs:
      command: 'custom'
      custom: 'sitecore'
      arguments: 'ser pull -i Content.Corporate

This will create the serialized items on disk needed as a basis for our resources file. Again, note that you only want to include the SCS modules that should end up in the resources file, in this case the Content.Corporate module only.

  - task: DotNetCoreCLI@2
    displayName: Sitecore itemres create
    inputs:
      command: 'custom'
      custom: 'sitecore'
      arguments: 'itemres create -o content -i Content.Corporate'

After running these commands, you will have created a resources file (following the examples it will be named content.dat) which can be copied to your Sitecore environment.
If your next step is to build a Docker image, you can immediately take this file and copy it to the App_Data folder for example.

Be aware

The earlier described process will allow you to create a synchronization process which can pull content from any environment and push it as part of your deployment.
There are however a couple of catches to this process. When you’re trying to synchronize large amounts of content, you will bump into certain (performance) limitations of Sitecore.

So far, we have seen that when you try to pull content from Sitecore, it will use a lot of its memory resources available, even to the point where it runs out of memory and Sitecore may restart.
To circumvent this issue, try and create SCS modules which have a limited scope of content it will pull. If possible, you can also decide to give your CM instance access to more memory. With the process we have in place, it seems that 6-8GB of memory is required for this process to run smoothly.

Another option you can try is to build in a retry policy to your serialization pull commands. You can do this in your Azure DevOps pipeline task my specifying the retryCountOnTaskFailure option as shown below.

  - task: DotNetCoreCLI@2
    displayName: Sitecore ser pull
    retryCountOnTaskFailure: 2
    inputs:
      command: 'custom'
      custom: 'sitecore'
      arguments: 'ser pull -i Content.Corporate'

Key take-aways

Building this content synchronization process has taught us a couple of things:

Sitecore’s more recent improvements and new tooling allows for greater flexibility in how you build features and use the Sitecore platform.
Sitecore’s new content serialization tooling offers a more flexible approach when it comes to pulling and pushing content from a Sitecore environment, which no longer requires files to be serialized on the disk of the environment itself.
The Item as Resources feature allows you to create a content package which is much faster to install compared to traditional Sitecore content packages.

Automated Sitecore content synchronization using Item as Resources

Creating a resource file in your deployment process

B﻿e aware

Key take-aways

Be aware