External Reference Key (E-KEY) matching behavior update

From 17.5 onwards, we are adding support for both AND and OR matching condition when multiple External Reference Keys are used to match a Product's External Reference. As a result, the fundamental logic for finding matching External Reference using E-KEY is completely reworked.

Enabling DSI constraints logic

The new DSI constraints logic will be enabled by default in the 17.5 Major release. The logic has been rewritten to better support both CET and Stage, allowing them to share the same backend for improved consistency. There should be no behavioral differences between the old and new constraints logic. However, if you encounter any broken or unexpected behavior after enabling the new DSI constraints logic, please report it to Dev Support. As a temporary workaround, you can override the forceDsApplyLogic() method in the DsPData or DsPDataProxy class to fall back to the old constraints logic.

class DsiPData

The following method is modified, as getting matching External References with matching E-KEY from a single option is no longer accurate.

Old: extend public ExternalRefType[] optionExternalsFromExtKeys(Option option, str{} usageTypes=null) {
New: extend public <ExternalRefType, str>[] optionExternalsFromExtKeys(str{} usageTypes=null) {

The following method is renamed:

Old: extend public Bool dbg_forceDsApplyLogic() {
New: extend public Bool forceDsApplyLogic() {

class DsiApplyConstraintsEnv

The following field and method in DsiApplyConstraintsEnv is removed as they are no longer used:

// Field
Removed: public str[] syncGroupIds;

// method
Removed: final public void appendSyncGroupId(str id) {

Changes to DataObj class uniqueKey

In 17.0 Minor, a new method collectExportUniqueKey() was added as part of the improvements to catalogue DB3 format saving. The main intentions were to :

• Collect uniqueKeys more efficiently and reduce memory usage.
• Generate a shorter and more accurate unique string identifier for DataObj. These changes has improved performance during the DB3 saving process.

However, this resulted in two different unique key implementations for each DataObj subclass. Hence, we are consolidating both methods into a single source of truth.

When is migration required?

Migration is required if you have a custom DataObj subclass (e.g., DsPData) that overrides the uniqueKey() method. The migration steps are:

• Move the uniqueKey generation logic from uniqueKey() to writeUniqueKey().
• Respect the includeClassId flag if the class is part of the unique key. It is recommended to use class.id as it is usually shorter than class.name or class.toS.
• Remove the uniqueKey() method from the class as it is no longer overridable.

Important note: Do not call uniqueKey() within writeUniqueKey() directly/indirectly and vice versa, as this will result in recursion.

If you encounter a compile error related to uniqueKey() calls, add the following use statement to package.cm.

use cm.abstract.dataSymInterface.catalog;

Interface changes related to uniqueKey

The following field in DsTableDataG2 class is renamed due to ambiguous naming:

Old: public str uniqueKey;
New: public str tableUniqueKey;

collectExportUniqueKey() in DataObj has been renamed to writeUniqueKey(), and is now an abstract method.

Old: extend public void collectExportUniqueKey(StrBuf b, bool includeClassId=true) { }
New: extend public void writeUniqueKey(StrBuf b, bool includeClassId=true) : abstract { }

For DataObj class and its subclasses:

• collectExportUniqueKey() is now renamed to writeUniqueKey().
• uniqueKey() is no longer abstract and is now final. It always generates and returns the unique key using writeUniqueKey().
• uniqueKey() method is removed in all DataObj subclasses

Old: extend public str uniqueKey() : null="null", abstract {
New: final public str uniqueKey() : null="null" {

// In cm/abstract/dataSymInterface/catalog/dataObj.cm:

    /**
     * Unique key.
     * A unique key is used to uniquely identify this object from other objects.
     * 
     * (It's known to be used to prevent duplicate append to SQLite databases)
     */
    final public str uniqueKey() : null="null" {
		static StrBuf buf();
		buf.clear();
		writeUniqueKey(buf);
		return buf.toS();
    }


// cm.abstract.dataSymInterface.catalog
public class DsiAddProductRefType extends DsBaseCodeType {
public class DsAliasType extends DsBaseCodeType : unstreamable {
public class ApplicationAreaType extends DsBaseCodeType : unstreamable {
public class DsBaseCodeType extends DataObj : abstract {
public class DsBaseExternalRefType extends DsBaseCodeType : abstract {
public class DsBaseNameType extends DsBaseCodeType {
public class DsCardConfig extends DsBaseCodeType : unstreamable {
public class DsClassificationRefType extends DsBaseCodeType {
public class DsClassificationType extends DsBaseCodeType : unstreamable {
public class DsConnectorType extends DsBaseCodeType : unstreamable {
public class DsConnectorRefType extends DataObj : unstreamable {
public class DsConstraintTable extends DsBaseTableG2 {
public class DsCustomOptionType extends DsBaseCodeType {
public class DsCustomInquiryType extends DsBaseCodeType {
public class DsCustomProdDataTable extends DsBaseTableG2 {
public class DsCustomTable extends DsBaseTableG2 {
public class DsVersionPolicy extends DataObj : unstreamable {
public class DsWire extends DsBaseCodeType {
public class DsExternalRefKeyType extends DataObj : inherit constructors {
public class ExternalRefType extends DataObj : unstreamable {
public class PrdExternalRefType extends ExternalRefType {
public class OptExternalRefType extends ExternalRefType {
public class DsiFeatureRefType extends DsBaseCodeType {
public class SFeature extends DsProductType : unstreamable {
public class DsiLayoutDomainType extends DataObj {
public class DsiLayoutType extends DsProductType {
public class DsLeadTimeProgramType extends DsBaseCodeType : unstreamable {
public class MaterialQtyType extends DataObj : unstreamable {
public class SMaterial extends DsBaseExternalRefType : unstreamable {
public class DsMeasureParamType extends DsBaseExternalRefType {
public class MeasurementType extends DataObj : unstreamable {
public class MtrlApplication extends DataObj : unstreamable {
public class NamedPoint extends DataObj {
public class DsNoteType extends DataObj {
public class OFDAHeaderData extends DsBaseNameType : unstreamable {
public class OFDAHeaderData extends DsBaseNameType : unstreamable {
public class DsiCodeRangeType extends DataObj {
public class DsiOptionRangeType extends DsBaseCodeType {
public class Option extends DsProductType : unstreamable {
public class PlacementOriginType extends DataObj : unstreamable {
public class PlacementType extends DataObj : unstreamable {
public class PriceTypeSeq extends DataObj : unstreamable {
public class PriceType extends DataObj : unstreamable {
public class PricelistType extends DsBaseCodeType : unstreamable {
public class ProductCatalog extends DsBaseNameType : unstreamable {
public class ProductLevel extends DsBaseExternalRefType : unstreamable {
public class DsProductType extends DsBaseExternalRefType : unstreamable {
public class DsiRangeType extends DsBaseCodeType {
public class DsRuleType extends DsBaseCodeType {
public class DsSectionConfig extends DsBaseCodeType : unstreamable {
public class DsTableDataG2 extends DataObj {
public class DsTableType extends DsBaseCodeType {
public class DsTableHeadType extends DataObj {
public class DsTableRowType extends DataObj {
public class UsageType extends DataObj : unstreamable {
public class VersionType extends DataObj : unstreamable {

// cm.abstract.dataSymInterface
public class DataCatalog extends DataObj : unstreamable {
public class DataCatalogs extends DataObj : unstreamable {
public class DsiPData extends DataObj {
public class DsiPDataOption extends DataObj {
package class DsiPDataCustomOption extends DataObj : abstract {

// cm.abstract.dataSymbol.ui.libraryBuilder
public class DsCustomLibrary extends DataObj : unstreamable {
public class DsLibBuildDataObjSeq extends DataObj {
package class DsCustomLibraryType extends DsLibBuildDataObj : unstreamable {
public class DsLibNewLDataObj extends DsLibBuildDataObj {
public class DsLibBuildShrinkType extends DsCustomLibraryType {
public class DsLibBuildMultiMemBtnType extends DsLibBuildDataObj {
public class DsLibBuildMemBtnType extends DsLibBuildDataObj {
public class DsLibDisplayType extends DsLibBuildDataObj {
public class DsLibImageDisplayType extends DsLibBuildDataObj {
public class DsLibDataCatalogType extends DsLibBuildDataObj {
public class DsSymbolInfo extends DataObj {
public class DsLibraryData extends DataObj : unstreamable {
public class DsLibrarySearchData extends DataObj : unstreamable {
package class DsLibraryFavLibData extends DataObj : unstreamable {

// cm.abstract.materialHandling.data
public class MhPData extends DsPData : replacing load {

The following debug flag is removed as it is no longer relevant after the uniqueKey changes:

In cm/abstract/dataSymInterface/database/dsSqlExport.cm:

Removed: public bool dsiOptimizeDb3Export() {
Removed: public void dsiSetOptimizeDb3Export(bool v) {

Remove uninstalled catalogs

The following interface changes have been made to facilitate the functionality to remove related catalog files when an extension is uninstalled.

cm.abstract.dataSymInterface.registerCatalogs.cm

Old: public void extensionRegCatDB3File(int cid, str dbFileName, str catCode,
					function(Object o) completeCB=null,
					function(Object o) outOfDateCB=null) {
New: public void extensionRegCatDB3File(int cid, str dbFileName, str catCode,
					function(Object o) completeCB=null,
					function(Object o) outOfDateCB=null,
					symbol sourcePkg=#:package : caller eval) {

Old: public void dsiRegisterEmbeddedCatalog(int cid, str dbFile, str{} offlinePrdCatalogs=null, bool offlineTurnOnAll=false) {
New: public void dsiRegisterEmbeddedCatalog(int cid, str dbFile, str{} offlinePrdCatalogs=null, bool offlineTurnOnAll=false, symbol sourcePkg=#:package : caller eval) {

Old: public void dsiRegisterEmbeddedCatalog(int cid, str dbFile, str{} offlinePrdCatalogs=null,
					    bool offlineTurnOnAll=false, bool ignoreSyncCycle=false) {
New: public void dsiRegisterEmbeddedCatalog(int cid, str dbFile, str{} offlinePrdCatalogs=null,
					    bool offlineTurnOnAll=false, bool ignoreSyncCycle=false,
					    symbol sourcePkg=#:package : caller eval) {

Also some interfaces were updated in the cm.abstract.dataSymbol and cm.extension packages as part of this change. Please refer to the documentation for those changes as well.

Performance improvement for loading db3 files in catalog creator

We have updated the sql import functionality in catalog creator to leverage more caching to improve load performance.

The following interfaces have changed to facilitate this.

class DataCatalogDBWrapper

Old: extend public void recreateCollection(DataObj oh, int ownerID, Field f) {
New: extend public void recreateCollection(DataObj oh, int ownerID, Field f, str[] bufList) {

Old: extend public void recreateUnaryColl(UnaryCollection coll, Field f, DataObj oh, str refTable, int ownerID) {
New: extend public void recreateUnaryColl(UnaryCollection coll, Field f, DataObj oh, str refTable, int ownerID, str[] bufList) {

Old: extend public void recreateMap(BinaryCollection m, Field f, DataObj oh, str refTable, int ownerID) {
New: extend public void recreateMap(BinaryCollection m, Field f, DataObj oh, str refTable, int ownerID, str[] bufList) {

class MtrlApplication

The _uniqueKey field in MtrlApplication was added to speed up unique key generation for certain environment. However, this field was also mistakenly streamed into DB3 format, resulting in unintended data entry being saved to the file. From 17.5 onwards, this field will be excluded from DB3 saving. The saved DB3 in 17.5 should remain compatible when opened in both older and newer versions of CET.

External Reference Key (E-KEY) matching behavior changes

The changes to External Reference Key (E-KEY) matching behavior are made for two main reasons:

• Introducing AND condition to align with the E-KEY behavior defined in the OFDA schema.
• Ensure OR condition is still achievable to avoid breaking existing catalogue implementation

Old Behavior

Prior to 17.5, the E-KEY matching was evaluated independently for each selected option, using an OR condition across all keys. If any E-KEY on a selected option matched an E-KEY on the Product's External Reference, it was considered a match.

New Behavior

The optionExternals and optionExternalsFromExtKeys methods in the DsiPData class have been changed so that the matching logic is evaluated using the combined set of E-KEYs from all selected options. The new matching logic is descriped as follow:

• AND across types: If an External Reference contains multiple E-KEY types, at least one E-KEY from each type must match.
• OR within a type: If an External Reference contains multiple E-KEYs of the same type, matching any one of those E-KEYs satisfies that type.
• Matching E-KEYs can come from different selected options. They are combined when evaluating whether the External Reference matches.

For example, given the following E-KEYs on an External Reference (E-KEY format: Type/Value):

E-KEYs in a Product's External Reference: X/A, X/B, Y/C

The matching condition is: (X/A OR X/B) AND Y/C

This means the selected options must contain either X/A or X/B, as well as Y/C, for the External Reference to match.

When is migration required?

Existing catalogues using multiple E-KEY types should be reviewed, as different E-KEY types are now evaluated using an AND condition. All E-KEY types defined on an External Reference must be satisfied by the selected options for it to match.

The migration work is mainly required on the catalogue side. For more details on how to migrate the catalogue, please refer to this page.

Remove uninstalled catalogs

We have updated the functionality for the "Keep catalogue files after uninstalling catalogue" option in the control panel under the "Catalogues" tab. Previously this functionality was intended to remove the files in the portfolio (cid) folder after all the catalogs in the portfolio were turned off (this state was known as the portfolio uninstalled state). However this did not work correctly.

We have fixed this functionality to work correctly now so that when the option is unchecked, all the catalogs under the portfolio have been turned off, the portfolio folder in the catalog directory will be removed now. Additionally when an extension is uninstalled when the option is unchecked, we remove all related portfolio folders that were registered by that extension. This removal is done silently and over time to minimize interruptions to the user experience.