A small fable of what is lost when web services are the default option for SAP integrations.

One of the earliest integration projects I took part in was a client’s effort to create a browser-based user interface for work execution on the factory floor. The previous user interface was a transaction in SAP GUI, which performed basic SAP production order confirmations, generated serial numbers, allowed workers to look up relevant work instruction documents and also registered some customer-specific details about production.

External interfaces for all those functions needed to be created in SAP. The list of integration technologies supported by SAP is fairly extensive (IDoc, RFC, OData, Web Services/SOAP, REST, TCP, MQTT) but HTTP in some form was more or less a given, since it was universally supported (unlike SAP RFC and OData), close enough to real-time (unlike message-based IDocs) and since the request-response integration model made things simple.

Of course, there are any number of ways to use HTTP, so just specifying “HTTP” doesn’t really nail down the relevant details. We had previously used OData for mobile app developments and Web Services for everything else, so we defaulted to using Web Services.

List of Production Orders

The browser UI needed to get production orders from SAP in order to display a list from which users could choose to execute one. Here’s the signature of the function module we came up with:

1
2
3
4
5

FUNCTION zget_production_order_list
  IMPORTING VALUE(im_caller)           TYPE string
  EXPORTING VALUE(et_production_order) TYPE zpp_t_ws_production_order
            VALUE(ex_success)          TYPE sap_bool
            VALUE(ex_error_message)    TYPE string.

The caller supplies an identifier (im_caller), based on which SAP knows what time period and which work centers to use for selecting production orders. When this function module is used to create a Web Service, the HTTP request + payload will look something like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

POST http://vhcalnplci:8000/sap/bc/srt/rfc/sap/zws_fms/001/zws_fms/zws_fms HTTP/1.1
Accept-Encoding: gzip,deflate
Content-Type: text/xml;charset=UTF-8
SOAPAction: "urn:sap-com:document:sap:soap:functions:mc-style:ZWS_FMS:ZgetProductionOrderListRequest"
Content-Length: 346
Host: vhcalnplci:8000
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.5.5 (Java/16.0.1)

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
                  xmlns:urn="urn:sap-com:document:sap:soap:functions:mc-style">
   <soapenv:Header/>
   <soapenv:Body>
      <urn:ZgetProductionOrderList>
         <ImCaller>BrowserUI</ImCaller>
      </urn:ZgetProductionOrderList>
   </soapenv:Body>
</soapenv:Envelope>

and the response would be:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

HTTP/1.1 200 OK
set-cookie: sap-usercontext=sap-client=001; path=/
content-type: text/xml; charset=utf-8
content-length: 517
accept: text/xml
sap-srt_id: 20220827/134152/v1.00_final_6.40/8BDB615CF8811EED89BE9CEA0142A2C2
sap-srt_server_info: NPL_001,21641 ,urn:sap-com:document:sap:soap:functions:mc-style,ZWS_FMS,ZgetProductionOrderList,6
sap-srt_server_info_ext: 0
sap-server: true
sap-perf-fesrec: 21678008.000000

<soap-env:Envelope xmlns:soap-env="http://schemas.xmlsoap.org/soap/envelope/">
   <soap-env:Header/>
   <soap-env:Body>
      <n0:ZgetProductionOrderListResponse xmlns:n0="urn:sap-com:document:sap:soap:functions:mc-style">
         <EtProductionOrder>
            <item>
               <ProdOrder>000000000001</ProdOrder>
            </item>
         </EtProductionOrder>
         <ExErrorMessage/>
         <ExSuccess>X</ExSuccess>
      </n0:ZgetProductionOrderListResponse>
   </soap-env:Body>
</soap-env:Envelope>

and should something go wrong in SAP (for example, an unhandled exception), the framework returns a response with 500 Internal Server Error status code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

HTTP/1.1 500 Internal Server Error
content-type: text/xml; charset=utf-8
content-length: 582
connection: close

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
   <soap:Body>
      <soap:Fault>
         <faultcode>soap:Server</faultcode>
         <faultstring xml:lang="en">RABAX occurred on server side</faultstring>
         <detail>
            <sap:Rabax xmlns:sap="http://www.sap.com/webas/710/soap/runtime/abap/fault/generic">
               <SYDATUM>20220827</SYDATUM>
               <SYUZEIT>134402</SYUZEIT>
               <ERRORCODE>MESSAGE_TYPE_X_TEXT</ERRORCODE>
            </sap:Rabax>
         </detail>
      </soap:Fault>
   </soap:Body>
</soap:Envelope>

Unfortunately, a response with 500 status code is also generated if one of the parameters is invalid (eg. text supplied in an integer parameter) or if a required parameter is missing, ie. in cases that should produce 400 Bad Request responses according to the HTTP standard:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

HTTP/1.1 500 Internal Server Error
set-cookie: sap-usercontext=sap-client=001; path=/
content-type: text/xml; charset=utf-8
content-length: 470
accept: text/xml
sap-srt_id: 20220904/102950/v1.00_final_6.40/B38B65D68F941EED8B846B886AA79115
connection: close
sap-server: true
sap-perf-fesrec: 148918.000000

<soap-env:Envelope xmlns:soap-env="http://schemas.xmlsoap.org/soap/envelope/">
  <soap-env:Header/>
  <soap-env:Body>
    <soap-env:Fault>
      <faultcode>soap-env:Server</faultcode>
      <faultstring xml:lang="en">Web service processing error; more details in the web service error log on provider side (UTC timestamp 20220904072950; Transaction ID 8280D1D62E110010E0063133882F3A00)</faultstring>
      <detail/>
    </soap-env:Fault>
  </soap-env:Body>
</soap-env:Envelope>

All and all, there’s not much more here than plain HTTP carrying a simple SOAP XML envelope, and the behavior is not much different either. Various complicated Web Service standards/additions (WS-*) exist and could add encryption, authentication, message routing and delivery guarantees to this basic exchange, but those are (in my experience) rarely used. Because the more complicated standards are avoided, the message structure is simple enough that it can be cobbled together and interpreted by hand, but of course the intended use is to generate an implementation automatically from a WSDL document.

From SAP point of view, a function module as a Web Service is convenient: because the framework takes care of XML parsing/transformation and parameter checking, most of the plumbing related to HTTP is automagic. Likewise for the caller, if their platform supports client generation from WSDL. No such convenience is (currently) available when working with plain HTTP in ABAP. What also complicates things is how (seemingly) harder it is to test plain HTTP services vs. function modules: function modules are testable by simply pressing F8 and supplying the parameters, whereas plain HTTP basically requires unit tests and supplying the inputs as XML/JSON payload. Knowing what I now know, I might opt for plain HTTP (since I don’t mind writing unit tests), but others might not even entertain the thought.

Leaky Interface/Abstraction

However, you can notice that a number of weird ABAP implementation details leak through the Web Service. The most obvious one is the way ABAP represents booleans (X for true and for false), which is quite unique among programming languages. It is possible to define function module parameters in such a way that booleans get represented “correctly,” but it requires that the programmer: (1) understands that forcing X instead of true on the callers is confusing and unnecessary and (2) knows that one must use data element xsdboolean to get a standard XML boolean. If one doesn’t spend time with the XML requests/responses, it is easy to miss this.

Another leaked detail are the prefixed zeroes in production order number (<ProdOrder>000000000001</ProdOrder>). In ABAP, certain fields always contain prefixing zeroes (a production order number is 12 characters long, ie. 11 zeroes and 1 one) but these fields are displayed in GUI without the zeroes (and vice versa, prefixing zeroes are automatically added to these fields when accepting user inputs). Unless extra steps are taken, clients would get these values with the prefixing zeroes. While this is not a catastrophe with IDs, which are mostly treated as opaque strings of characters, it does look goofy when displayed to users (and forcing all clients to implement special string handling for discarding prefixing zeroes is also silly).

Lastly, because the Web Service gets parameter names directly from the function module, ABAP parameter prefixes IM_, ET_ and EX_ are also visible in the service definition. I’d argue that Caller would be better than ImCaller.

These all trace to the same root cause: the fact that the service definition is tightly bound to the function module. This means that unless special actions are taken, implementation details leak to the caller. However, if special actions are taken, they may render the function module unsuitable for internal use (in our example, stripping prefixing zeroes from production orders would make these ids invalid). In this vein, Webber et al. (2010, 382–385) note that unless proper care is taken, this way of creating Web Services makes it possible to expose functionality too deep in the Domain Model. Ie. if development artifacts are not created especially as external interfaces, but instead internal functions are reused for external interfaces, the resulting interfaces expose unnecessary details and are also unstable.

We were mostly successful in not leaking these details to the caller and in keeping our Web Service function modules separate and stable, but one major thing was left out of our interface: caching.

Native HTTP Caching

Although SOAP messages are carried over HTTP, just a limited subset of HTTP features are used. Only POST method is used. If the Web Service was generated from a function group, all services share one URI and the resource is only communicated in the request body. Most crucially, because only POST HTTP method is used and HTTP header fields are not part of the picture, native HTTP caching is unavailable.

Caching in some form or another was a must. Reading production orders to create the list that would be returned to the caller could take a little while, depending on how busy the work center was and how long a time frame was used. Thus to allow the caller to poll the list as they liked without negatively affecting SAP performance, we generated the lists by running a background job every 15 minutes and saving the results. This allowed us to just return the already generated list to the caller without reading the production orders again.

There is a little wart in that last sentence. We always returned the list, whether it was big or small, changed or identical to the last one we had returned. Our function module did not include or define any method for what in HTTP would be called “Conditional requests.”

In HTTP, the server can return an “ETag,” an identifier for a particular version of resource. Then, when the client requests that same resource again, it can include that identifier in a If-None-Match header field to make the request conditional.

1
2
3
4
5
6

HTTP/1.1 200 OK
etag: "v1"
content-type: text/xml; charset=utf-8
content-length: 10000

...
1
2
3
4
5

GET /productionOrderList HTTP/1.1
if-none-match: "v1"
accept: text/xml

...

If the resource has not changed, the server can respond with a 304 Not Modified status code without sending the list of production orders over again, allowing for more lightweight polling. (Webber et al. 2010, chapter 6; HTTP Semantics, section 13)

1
2
3

HTTP/1.1 304 Not Modified
etag: "v1"
content-length: 0

What we could have gotten for free with native HTTP, our function module did not define. Though then again, what we got for free with Web Services (automatic XML handling), we would have needed to implement by hand. And this assumes that the browser UI would have been a sophisticated enough HTTP client to have benefited from conditional requests. I guess what ultimately bothers me was how POST was used for an operation with more or less exactly GET semantics.

Tool Cohesion

I mentioned at the start that the list of production orders was only one of the services required by the browser UI. This means that decisions as to the technical details of this particular service are not completely independent of choices made elsewhere.

One constraint is the number of tools needed to monitor and debug these services, and the broad familiarity with them: breaking one or more away as REST services would double the number of transactions used for interface debugging/monitoring. If all services are Web Services, we can just set trace for the browser UI user in transaction srt_util and see everything that goes on between the two systems. If one or more services are not Web Services but REST Services, we also need to set a recorder in transaction SICF. Technically we could record all HTTP requests the browser UI user sends in SICF, thereby also capturing the Web Service requests and achieving full visibility, but that’s not most people’s first instinct when working with Web Services. In my experience, seeing the HTTP Request is often enough to debug a Web Service call, making SICF recording completely appropriate when working through the kinks of an interface. But I won’t always be there to solve issues, so others and established ways of working must be considered.

Production Order Events

This section is mostly unrelated to the interface exposed by SAP. Instead, it considers how the list of production orders could have been kept current using techniques other than what we actually used.

I mentioned that we ran a background job to create the list of production orders that we’d return to callers. The list contained only open production orders and was mechanically re-created using the work center and time period parameters when the background job ran. Because the job ran infrequently, it didn’t matter much that the number of production orders read might make the runtime somewhat long.

This setup of course meant that after a production order was changed, it could take up to 15 minutes for this change to be reflected in the production order list. Given that production orders reflect and represent actual material reality on the factory floor and because production schedule must be planned and changed with more than 15 minutes of lead time in order to allow material movements to production area to happen, this was not a deal-breaker.

The alternative to mechanically re-creating the lists according to some specified intervals is of course, re-creating the lists when changes happened, ie. we’d need to be notified that something happened to one or more of the production orders. In other words, subscribing to events related to production orders.

It turns out that something like events does exist (or can be simulated) in standard SAP. It also turns out, as is often the case, that there are multiple, partially overlapping implementations of this concept:

  1. Business Object Events (transactions SWO1/SWE2)
  2. Message/Output Control (transaction NACE)
  3. ALE Distribution (transactions BD54, BD64, WE21, WE20, also report RCCLORD)
    • Using Message/Output Control
    • Using Change Pointers (Change Documents)
    • Running report manually or as a background job
  4. User Exits / Enhancements

Business Object Events

Business Objects are intended to act as an interface between SAP Workflow and, well, business objects like purchase orders, sales orders and production orders. In addition to supporting data retrieval and various operations, Business Objects can define events which workflows can then react to (eg. a purchase order approval workflow should reset if changes are made to the purchase order). These events can also be used outside of workflows via transaction SWE2, where arbitrary function modules and class methods can be defined as subscribers for events. (Dart et al. 2014, chapters 15, 16 & 18)

Business Objects understandably keep these events at the level of business logic, rather than the lower technical level. To wit, events defined by the production order Business Object:

Production Order Business Object

Released, FinallyConfirmed, TechnicallyCompleted, Deleted are all significant events from the point-of-view of business processes. What’s missing is a simple technical Changed event, but Business Objects are not technical objects. This fact renders this version of events unsuitable for (re-)creating our list of production orders, since we can only subscribe to a fairly limited assortment of state changes.

Also, depending on the configuration of the event link (in SWE2), the event can trigger immediately or it can be queued/batched for later processing. The choice alters the timing of the event from somewhat real-time to delayed until a background program is used to process event queues. It also affects the error-tolerance of event handling, with event queues being more robust. (Dart etc al. 2014, chapter 18)

Change Pointers

What in SAP serves as a better proxy for plain technical change events are Change Pointers, which rely on Change Documents. Like is often the case in systems based on a relational database, what one finds in VBAK, VBAP, AFKO, AUFK and AFRU tables is a snapshot in time. Without some extra mechanism, the history of these table is lost when data is mutated (as opposed to an event sourced systems, where history can be replayed) (Ghosh 2016, chapter 8).

Change Documents are a generic mechanism in SAP for adding history to a table or a group of tables. When used, they produce an audit log of changes, ie. what changed, how, when and by whom. Change Pointers are an additional mechanism built on top of Change Documents: when configured, Change Document creation also triggers the creation of Change Pointers, which are basically a To-Do list based on which another program will then distribute master/transactional data changes according to the ALE distribution model. (Kasturi 1999, chapter 5; Perfiljeva 2016, chapter 3)

Although we were not trying to proactively distribute production orders to the browser UI system, as that was not the chosen integration scenario, we could have distributed the messages locally and used a function module port (transaction WE21, ABAP-PI port type) to trigger the production order list update (Kasturi 1999, chapter 5; Hadzipetros 2014, chapter 7; Perfiljeva 2016, chapter 2 & 3).

Like queued Business Objects events, Change Pointers are triggered only when the processing report is run (Kasturi 1999, chapter 5; Perfiljeva 2016, chapter 3). Ie. Change Pointers are by default batched and it’s not possible to configure them to be real-time.

However, the fact that, without additional configuration, production orders are simply not covered by Change Documents renders this option unsuitable for our production order list.

Message/Output Control

Message/Output Control is a mechanism which allows various outputs like printouts, physical mail and email to be triggered according to flexible conditions. Among the assortment of possible outputs are also Workflow events, EDI and ALE message distribution. Because Message/Output Control is freely extensible with custom output types, which are implemented as standard SAP reports, only human imagination and bravery limit what can be done with it. Message timing is also configurable, with both immediate and batched options available and configurable on the output record level. (Hadzipetros 2014, 276–296; Williams 2008, 375–382; Perfiljeva 2016, chapter 3)

What makes Message/Output Control unsuitable for our production order list is simply that production orders don’t use it.

ALE Distribution

ALE Distribution allows master and transactional data to be distributed to external systems. ALE (short for Application Link Enabling, a decidedly unhelpful and confusing name) relies on “distribution models,” ie. a specification of which external system wants to receive what logical messages from the current SAP system.

For example, a Manufacturing Execution System (MES) for a certain plant could be interested in only the production orders of that specific plant. A distribution model would then be setup such that the logical system representing the MES system would receive production order logical messages (message type LOIPRO), filtered to only the relevant plant. Then when a program is run to distribute production order data, production orders are translated into IDocs and the ones matching the plant filter criteria are sent to the MES system. During processing they may be transformed from one basic type to another and are ultimately passed to a port according to the partner profile of the MES system. (Kasturi 1999, 11–12, 26–27; Perfiljeva 2016, chapter 3)

Depending on how the process is configured, this is just blunter version of what Change Pointers do. Triggering a report to distribute messages according to the distribution model is more or less the same as just reading production orders from the database, except that involving ALE & Idocs in this way brings with it many complications with little to no benefits. An enhancement could be created to trigger ALE distribution of production order on save, but that’s of course no longer standard SAP.

While we’re on the topic, a word on the links between IDocs, EDI and ALE. Unless you’re already some roads into the topic, it’s hard to find a good explanation laying the three out in an understandable way. Perfiljeva (2016, 21) notes that many books don’t even attempt this and instead just dodge the question, which is also more or less my impression.

How I personally came to peace with these three terms was to first set aside IDocs: both EDI and ALE messages appear as IDocs (Intermediate Documents) in SAP; IDoc is the common structure/container for the data in these messages and partner profiles and port definitions are also used by both. Then the remaining practical difference between the two is how the receivers are chosen: ALE, as described earlier, relies on the distribution model to decide who gets what message. Ie. a message is not really intended for anyone in particular, but instead will be sent to anyone who has indicated their interest via the distribution model. In contrast, EDI messages are always intended for a certain receiver. A specific customer gets the confirmation for a sales order, a specific vendor is sent a purchase order for materials they sell. These messages are then transferred between the two companies, maybe as one of the traditional EDI formats (EDIFACT/X12), maybe in some other form (EDI does ultimately mean Eletronic Data Interchange, specifying no transfer technology or message format explicitly).

I think this split is also reflected in how the function module MASTER_IDOC_DISTRIBUTE is used:

1
2
3
4
5
6

CALL FUNCTION 'MASTER_IDOC_DISTRIBUTE'
  EXPORTING
    master_idoc_control        = ls_master_idoc_control
  TABLES
    communication_idoc_control = lt_idoc_control
    master_idoc_data           = lt_idoc_data.

master_idoc_control parameter accepts IDoc header data, one of part of which is the receiver. If this part is left unspecified, ALE distribution model is used to determine the receivers. Somewhat oddly this is not documented in the function module documentation, but message 003(B1) says as much:

Message 003(B1)

User Exits / Enhancements

The naive/obvious way to receive notifications of production order changes is to search for an appropriate User Exit or other enhancement option which would allow the insertion of code that would trigger the re-creation of the production order lists when a production order was saved. This could take the form of an asynchronous or transactional function module call or a BP_EVENT_RAISE function module call to trigger a background job.

But these kinds of special-case exits and enhancements are what turn SAP system codebases rancid as special cases and special logic accrue over years and decades and as both the original developers and the original business users move on. Sometimes you don’t have a better (standard) option, but one should not be too eager to reach for this option.

Sources

  • Dart, J. – Keohan, S. – Rickayzen, A. 2014. Practical workflow for SAP. Galileo Press. 978-1-4932-1009-1.
  • Ghosh, D. 2016. Functional and reactive domain modeling. Manning Publications. 978-1-61729-224-8.
  • Hadzipetros, E. 2014. Architecting EDI with SAP IDocs: the comprehensive guide. Galileo Press. 978-1-59229-871-6.
  • HTTP Semantics. <https://www.rfc-editor.org/rfc/rfc9110.txt>, fetched 2022-09-12.
  • Kasturi, R. 1999. SAP R/3 ALE and EDI Technologies. McGraw-Hill. 978-0-07-134730-3.
  • Perfiljeva, J. 2016. What on Earth Is an SAP IDoc?. Espresso Tutorials. 978-1-5237-9740-0.
  • Webber, J. – Parastatidis, S. – Robinson I. 2010. REST in Practice: Hypermedia and Systems Architecture. O’Reilly. 978-0-596-80582-1.
  • Williams, G. C. 2008. Implementing SAP ERP sales & distribution. McGraw-Hill. 978-0-07-149705-3.