Key points of Sitecore CDP and Sitecore Personalize

-

I would like to share some key points of CDP and Personalize.

For preparing for the certification exam also these key points might help along with Sitecore documentation of both CDP and Personalize.

From the Sitecore e-learning portal, Sitecore CDP & Personalize Technical Deep Dive topic provides the best training for CDP and Personalize.

From Feb 2023, the Engage JavaScript Library and Engage SDK are introduced, these are modern, easy-to-use tools that help us integrate the application with Sitecore CDP.

Based on my learning I will share some important points on each concept in CDP and Personalize.

  • Sitecore CDP and Personalize basics
    • There are 2 types of API versions.
      • 2.0 (Old Boxever)
      • 2.1(Sitecore CDP and Personalize)
    • CDP and Personalize use these below to get behavioral data, provide personalized content, etc
      • Stream API (lets you send real-time behavioral and transactional data about the users of your application to Sitecore CDP).
      • Batch API(lets you upload large amounts of guest data and offline orders efficiently to Sitecore CDP).
      • Rest API(lets you retrieve, create, update, and delete data that is available in Sitecore CDP).
      • Data lake export service(lets you access, export, and transfer all your organization's data).
    • Data extensions limits
      • The guest data extension limit is 100 attributes.
      • The order data extension limit is 50 attributes.
      • The event data extension limit is 50 attributes.
      • Data extensions are also allowed for Order items.
      • 6 data extension groups are allowed for guests.
      • The data extension recommended name is ext and the key is the default.
      • Only one data extension per event type is supported.
      • Data extension format
        • "extensions": [
        {
            "name": "ext",
            "key": "default",
            "refundable": true
        }]
    • All the data captured in CDP is by Events
      • Stream API creates events using the following API 
      • Browser API: /v1.2/browser/create.json?client_key=<CLIENT_KEY>&message=
      • Events API: /v1.2/event/create.json?client_key=<CLIENT_KEY>&message=
      • Stream API supports synchronous calls and sends write-only data
      • All other Rest APIs will use /v2
    • TMS integration
      • The first step is to load the .js library of Boxever or Engage javascript library.
      • A drawback of using TMS will result in a flicker issue.
    • Native mobile devices will use direct HTTP requests to integrate with CDP/Personalize.
    • The main 5 entities are Guests, Order, Order items, Events, and Sessions.
    • IDENTITY events can be triggered multiple times in the single browser session
    • Stream API will use PII details to trigger the IDENTITY events using the Identifiers array, direct passing is not allowed, but it is allowed in Rest API.
    • Identity Resolution will be done in hierarchical order if multiple identity rules are defined.
    • Batch API is used to do Offline identity resolution.
    • The VIEW event is the only event type you can use to capture tracking data.
    • Data extension of Guest will be found in the Properties tab of the Guest dashboard·       
    • Debug will be enabled by the Features menu.
    • Sitecore Personalize automatically creates an ID when we name the experiment or experience, if we create the same name (Recommend Offer) of the experience, or experiment that we deleted before Sitecore Personalize will append a number to the ID Example: Recommend_Offer_2 
    • Once the guest is Retired we can't see it in the dashboard or access it through any API.
    • Sitecore Personalize retains the last 40 sessions within the last 90 days.
    • If any of the last 40 sessions are offline, Sitecore Personalize only retains a maximum of 10 offline sessions. For example, if a guest's last 40 sessions consist of 12 offline sessions and 28 online sessions, Sitecore Personalize only retains 10 offline sessions.
    • Events are limited to 100 from 40 sessions and 20 orders.
    • Use ?expand=true to expand the details from entities, we can directly expand subresources by ?expand=subresourcename
    • Sending Order with OMS or with cookies / local storage (ORDER_CHECKOUT)
    • Sending Orders without OMS and no cookies/local. storage(ADD,ADD_CONTACTS,CONFIRM and CHECKOUT)
    • Session data can be captured in the VIEW event.  Example

                    "sessionData":{  
                                              "deep_link":"https://{domainname}/search?product=shoes",
                                              "is_logged_in":true,
                                              "assistance":false
                                           }

    • The mandatory parameter in CONFIRM event is an array of product  item_id.
    • CLEAR_CART event to ensure that Sitecore CDP ignores any products or contacts that have been passed to Sitecore CDP during the browser session.
    • The audience template will be used in Real-time audience targeting, it will allow adding form fields to help marketers easily to configure Real-time audiences with the need to know javascript/CSS, etc.
    • JS Module is used to have a reusable piece of code, the maximum size is 100kb, and a call to external service is not allowed.
    • An offer template is used to create offers with the help of defining structured data attributes for presenting to the guest. Attributes types are (string Number and Date).
    • Web Template is used to enable marketers to get data from the users, publish some notifications, etc. Web Template is also used to enable developers to add Forms fields, or to use by marketers without Forms.
  • Segmentation:
            • The batch segment enables us to build and query the guest based on the defined segment criteria.
            • The segment takes 24 hours to build.
            • Segment entities are Guest, Event, Session, Order, and Order Item.
            • Once Segment rules are defined using Calculate Audience button to check the number of guests available for the rule defined, Usually, we will only receive guest_ref as a response from Segment.
            • Set Limit filter enabled to limit the guest to be included in the segment with different conditions.
            • 2 Modes are available to define segments ( Basic and Advanced).
          • Audience Sync:
            • Audience Sync is used to export segment data from Sitecore CDP to an external system for campaigns, Social media, reporting, etc.
            • Steps followed to create Audience Sync
              • Create External Services destination
              • Audience Sync Template
              • Audience Sync
              • Segment
            • Data is exported in 2 formats based on the user setting (JSON / CSV)
            • Audience Sync can be triggered manually or scheduled
            • Necessary to execute Audience Sync (flowRef,segmentRef, and datasetDate)
            • Status("QUEUED", "PROCESSING", "SUCCESS", "FAILED", "PENDING", "CANCELLED")
          • Experiments:
            • This is used to do A/B testing.
            • Status(Draft,Publishing(Scheduled,Live,Draft,Pause),Live,Schedule,Paused,Completed)
            • The statistics on the Performance tab update every 24 hours.
            • Statistics will be displayed in the Operation Tab in 15 seconds.
            • Optimized testing will be enabled, and if enabled the individual variant traffic allocation will disappear, as variants will be dynamically allocated with the help of a muti-armed bandit algorithm.
            • Traffic allocation to variants will be done based on the buckets, there is a total of 120 buckets available. if we have 2 variants we can assign some buckets to one variant and the remaining to another variant.
            • Targeting the audience is done by Segments and Real-time audience (with the help of an audience template), Both can be used at the same time.
            • Real-time Audience or the Segment are not used for traffic allocation to the experiment / Variant.
            • Real-Time Audiences use real-time session information to target the audience(for example: searching for flights/loan applications etc) unlike segments that build every 24 hours.
            • Web Experiment:
              • QA Tool used to preview web experiment. and helps to change the control variant and check if any errors, also to switch the Production flows to validate the experiment.
              • This kind of experiment uses Page targeting options to execute the web experiences.
            • Triggered Experiment:
              • This experiment will be get triggered on any defined events
              • Triggered Experiment requires Destination Connections to be created to send notifications etc.
              • If we need to send the notification email and SMS on the abandoned cart, we need to create two triggered experiences one for SMS and the other for the Notification email.
              • Triggers are mainly used to execute web experiments/experiences on any events that happen either out of the box or custom events.
            • Interactive Experiment:
              • Interactive experiments enable developers to run tests across their full technology stack while serving dynamic content and offers.
              • Friendly Id is used to trigger this experiment/experience from Stream API/Interactive API.
            • Metrics Calculation:
              •  If we lower the confidence level, the required minimum sample size to reach statistical significance reduces.
              •  if we increase the confidence level, the required minimum sample size to reach statistical significance increases.
              • The experiment will still be running due to the minimum sample size not being reached.
            • Goals:
              • Sitecore Personalize automatically calculates the minimum sample size required to meet statistical significance, based on the chosen primary goal.
              • Primary Goal used to decide the Variant winner.
              • Multiple secondary goals can be created to track more details.
              • Different types of goals (Revenue Goal(Binary and Continuous), Exit, Bounce Rate, Page View, Custom Goal(Binary and Continuous).
          • Experience:
            • It is similar to Experiments but not used for A/B testing. Once A/B testing is done with Experiment, then that will be created as Experience to personalize/provide offers, etc.
            • Variants will not be created by default like in an experiment.
          • Decisioning:
            • The decision model is used to provide the best offer to the customers based on the business requirement by defining decision variants.
            • Decision model States ( Draft, Test, Production, and Archive).
            • A total of 8 variants can be created in the Draft state without archived.
            • A total of 6 variants can be placed in the production state.
            • A silent Test will be done in the Test state if the decision model is assigned to experience.
            • If more than 2 variants are available in the Production state, then audience size can be updated for individual variants, if only one variant is available then 100% allocation is assigned automatically.
            • The output of one decision table will become the input of another decision table with the help of Output Reference.
            • Offers will be based on the offer template.
            • Decision model components ( Input Data, Decisions, Knowledge Sources, and External Systems).
            • Offers won't be displayed in the result if the Knowledge Source is not connected to the Decision Table.
            • The decision model can be re-used across multiple experiments and experiences without making any changes.
            • Each variant can be easily rolled back if there is any issue with the help of Revisions.
            • Multiple decision table is allowed to be created in the single decision model.
            • A variant in Test / Production / Archive cant be edited, If the Variant from the Test state is moved to the Draft state then all the Slient Test data will be get lost.
          • Batch :
            • You can use the Batch API to upload guests, orders, and tracking events in bulk offline.
            • The batch upload will happen asynchronously.
            • If the uploads are complete user can view the details immediately in the Sitecore CDP without waiting for 24 hours.
            • The file type supported for batch Api is gzip(.gz).
            • The record in the file should be a JSON record per row.
            • Batch Schema (guest, order, tracking, and migration), modes are (insert, upsert guest).
            • The file size limit for Batch upload is 50MB.
            • MD5 Checksum on gzipped file.
            • Using Basic Authentication.
            • Create a random Guid (batchRef) and use it in the URL to get the location to upload the batch file.
              • PUT /v2/batches/batchRef 
              • the response contains the batch upload location path from the attribute location.href 
              • The uploaded file will be available for 1 hour only.
            • Use the above URL to upload, using Base64 on the MD5 checksum done previously and using AES256 encryption for x-amz-server-side-encryption.
            • Status(uploading,processing,success,corrupted and error)
            • Guest profile migration from one guest profile to another guest profile can be achieved. using the batch API's schema migration and mode guest. Migrating guests will also migrate Sessions, Orders, and Flow executions.
          • DataLake Export Service:
            • Sitecore CDP export data lake service runs daily. 
            • We can copy all organization data from the Sitecore CDP data lake to Amazon S3. also from Amazon S3 to another Amazon S3 bucket or on-premises.
            •  Entities exported (guests, events, sessions, order, order items, and experience_definitions)
            • To verify the status of exported status by verifying the _SUCCESS file created in the export Amazon S3 folder.
            • The exported file format is Parquet.
            • Data is formatted in the YYYY_MM_DD format.
            • Each time the Data lake export service runs it fully rebuilds historical data from guests, orders, order items, and experience_definitions, but it rebuilds only the last 3 days data for sessions and events, events before 3 days are not rebuilt but included in the export.
          • Connections:
            • 3 types of connections are available
              • Data System(used for getting data from external applications through API . used in Programmable a decision model to take decisions based on the external data).
              • Destination(used to push data to an external destination by API, used in Triggered experiment/experience).
              • AI(for machine learning based on the data).
            • 3 types of authentication available(OAuth 2, Basic and None).
            • Adding query parameter https://example.com/api/${email}.
            • The default Connection and Read timeout is 1000ms.
          • Testing / Troubleshooting:
            • QA Tool is used in Web experience/Web experiment.
            • The Preview API tool is used to verify the API Response from the Interactive experience.
            • Test Canvas is used to verify the API response / Validating offers etc in the Decision model.
            • Test Connection in Triggered experience.
            • Browser developer tools to verify if events executing properly, are also used to verify if the CDP javascript library loaded or not.
            • Triggering events from the browser console is easy with Browser developer tools.
            • In order to preview web experiment/experience web_flow_target should be updated in the javascript library.
            • Events not appearing in the Sitecore CDP/ Personalize because of invalid PointOfSale or it is missing.
            • Postman is not used for troubleshooting.
          • Error Codes:
            • Web experience /experiment throws 400 if identity information is missing.
            • 401 if authentication failed, mostly because of an invalid Clientkey.
            • If the type attribute is missed then it will result in 400 error.
            • if the browser_id attribute is missed or the target is wrong then it will result in a 404 error.
            • In Batch API  if Checksum is altered then results in 409 error.
            • In the Batch API file Size does not match the size passed in the body then resulting in a 400 error.
            • If the Content-type is missing in Batch API request then will be getting 401 error.
          • Programmable:
            • This is used to run server-side javascript to exact and filter data from entities, and external systems.
            • This is also used to access the custom attributes to process offers accordingly.
            • This can be exported or downloaded into a .txt file for use in other decision models.
            • Programmable will make use of the JS Module to load the snippets created.
          • Useful API:
            • Guest search based on PII
              •   https://{apiEndpoint}/v2/guests?email={test@test.com}
            • Guest search based on the identifier
              •   https://{apiEndpoint}/v2/guests?identifiers.provider={provider}&identifiers.id={id}
            • Guest data extension
              • v2/guests/guestRef/extext
            • Trigger experiment/expereience
              • v2/callFlows
            • Event Create
              • https://api.boxever.com/v1.2/event/create.json?client_key={clientkey}&message=
            • Browser Create
              • https://api.boxever.com/v1.2/browser/create.json?client_key={clientkey}&message=
            • Get Batch 
              • v2/batches/batchRef
            • Get Audience Sync Batch Job
              • /v2/batchFlowsJob/batchJobRef
            • Get Audience Sync Batch Output
              • /v2/batchFlowOutput/batchJobRef/files
          • Important Events

          VIEW

          Captures guest's page view details.

          IDENTITY

          Identity the guest based on PII or non-PII using Identifiers.

          CLEAR_CART

          Clearing the current cart.

          FORCE_CLOSE

          Abandon the session.

          ADD

          Adding the products to the guest cart.

          ADD_CONTACTS

          Adding the contact details for getting the order details.

          CONFIRM

          Confirm the products purchased.

          CHECKOUT

          Checkout event for products purchased.

          ORDER_CHECKOUT

          Checkout event for the products purchased used for OMS or when cookies and local storage are available.

          SEARCH

          Getting the search details of the guests.

          PAYMENT

          Capturing payment information of the order, and credit card details will not be captured and this is optional.


          • Comparison with Boxever and Engage Library

          Boxever Javascript library

          Engage Javascript Library

          Javascript version 1.4.9

          1.0.0

          type and browser_id  attributes are mandatory in sending Event requests

          Not needed

          pos , target attribute used

          PointOfSale,targetURL attribute used

          Example US region
          app-us.boxever.com


          api-engage-us.sitecorecloud.io

          To verify if the library is loaded in the application use _boxever

          To verify if the library is loaded in the application use engage

          Trigger View Event

          const viewEvent={

          …

          type=”VIEW”

          }
          Boxever.eventCreate(

          viewEvent,

          response => console.log(response),

          "json"

          );

           

          Trigger View Event

          engage.pageView(eventdata,dataExtension)

          Trigger Identity Event

          const viewEvent={

          …

          type=”IDENTITY”

          }
          Boxever.eventCreate(

          viewEvent,

          response => console.log(response),

          "json"

          );

           

          Trigger Identity Event

          engage.identity(eventdata,dataExtension)

          Trigger all other Events (ADD,CHECKOUT,CONFIRM etc)

          const viewEvent={

          …

          type=”ADD”

          }
          Boxever.eventCreate(

          viewEvent,

          response => console.log(response),

          "json"

          );

           

          Trigger all other Events

          engage.event(“ADD”,eventData,dataExtension)

          Trigger Experience / Experiment

          const flowObject ={

          …

          friendlyId=”experienceId”

          }

          Boxever.callFlows(

              flowObject,

              response => console.log(response));

           

          Trigger Experience / Experiment

          const personalizationData ={

          …

          friendlyId=”experienceId”

          }

           

          engage.personalize(personalizationData);

          Adding events to queue and execute the event

          _boxeverq.push

          Adding events to queue and execute the event

           

          engage.addToEventQueue("CONFIRM", eventData);

            engage.processEventQueue()

           


          Boxever.triggerExperiences()


          engage = await init({   

                webPersonalization: true   

            });

          engage.triggerExperiences();

          Boxever.getID()

          engage.getBrowserId()

          Boxever.addUTMParams(event)

          engage = await init({         includeUTMParameters: true,   

            });

          Getting Guest reference

           Boxever.browserShow(...)

          		Getting Guest reference
          engage.getGuestId()

          Comments

          Post a Comment

          Popular posts from this blog

          Custom Item Url and resolving the item in Sitecore - Buckets

          -
          Image
          I have a situation to update the Item URL to be SEO friendly as my items in content tree looks like below. In the above image in order to resolve the item, Sitecore will find the item from the below URL http://domainname/en/dine/nv/chicken but I need to form a URL that is more SEO friendly like http://domainname/en/chicken In Sitecore by Out of box, we can achieve this by using Aliases , we can create an Aliases under System à Aliases and in the Linked Item field select the target item. But this is very tedious as we need to go to every item in the content tree and create Aliases, instead of that we will be going to do it programmatically using custom pipelines. ·          Link Provider (Provides the custom URL for the item based on the templates) ·          Item Resolver (Resolves the item based on the custom URL generated by CustomLinkProvider , that we going to create later...
          Read more

          Fixing Sitecore Buckets folder path - Items created after 12 AM server time zone

          -
          Image
          We are working on a 24/7 content publishing site, so it is important to have articles published on the correct date and time.  We are using Sitecore buckets to manage article items. In the Sitecore content tree, when the content author tries to look at the items created after 12 AM server time, the items are created under the previous day folder only,  but it is supposed to be created in the current day folder( ie. new bucket day folder to be created and the items to be added in this new day folder ). We have hosted the site in Azure VM in the UAE region. UAE is 4 hours ahead of GMT/UTC (UTC+04:00) so items created between 12 AM to 4 AM are always created under the previous day's folder, but when the item created after 4 AM is properly created under the current day folder in the content tree. Example: If the item was created on 2024 à 09 à 24 between 12 AM to 4 AM then still the item is made under the bucket folder of 2024 à 09 à 23 The created date and the updated ...
          Read more

          Sitecore Search - API Crawler with Edge Pagination

          -
          Image
          Sitecore Search provides different ways to index the items from 3rd party systems with the help of Source Connectors. There are different types of Source Connectors available: API Crawler API Push Feed Crawler Web Crawler Web Crawler(Advanced) I have provided more details here about the different sources for crawling different types of documents to get indexed into Sitecore Search. In this article, I would like to share the details about API Crawler in more real-time querying data and adding to the index. We have a scenario where I need to index all the items along with their placeholder components based on a specific template that has a layout assigned to it. I decided to utilize the Experience EDGE GraphQL Query to get the data and index it to Sitecore Search. We can use this for non-layout items as well. This article covers the following: GraphQL - Layout Query with Components GraphQL - Item Query GraphQL - Item Query with Pagination GraphQL - Search Query with Pagination Graph...
          Read more