pofox : blog<code>

08 August 2015

BizTalk Certification

I'm happy to announce that I sat the exam for the Microsoft Partner Network BizTalk Technical Competency Assessment for Application Integration and passed. I'd like to make mention of some resources I felt really stood out in helping me prepare for the exam:

BizTalk 2010 Patterns by Dan Rosanova is based on BizTalk Server 2010 but a lot of the fundamentals still apply.

Microsoft BizTalk Server (70-595) Certification and Assessment Guide (2nd edition) by Johan Hedberg, Kent Weare, Morten la Cour has been updated to include BizTalk 2013 and as far as I know the only study guide that covers the latest release of BizTalk.

05 May 2011

Oracle Adapters and BizTalk

Here's a quick list of steps to do when you need to call a Oracle stored procedure from BizTalk (Assuming you have already created the Oracle schema, tables and stored procedures):

In Visual Studio, right click on your BizTalk schemas project and choose Add > Add Generated Items
Choose Consume Adapter Service
Select OracleDBBinding from the binding drop down
Configure the URI either directly or through tnsnames.ora and enter the username/password
Click on Connect. If everything is configured correctly, the category list will be populated from Oracle
Under your chosen schema, select Procedure
In the Available categories and operations window, select the procedure(s) you need to call and add them
The Filename Prefix is the string put in front of all the generated schema files. Enter an appropriate one and click OK
It will generate the xsd schemas for the request and response messages. Promote whatever fields you need now
It will also generate a binding file which you can import into your BizTalk application to save yourself from having to configure that stuff manually
Once you've imported the bindings, you will see a newly created send port with a name like WcfSendPort_OracleDBBinding_...
Configure the WCF-Custom transport properties. Under the Credentials tab, either specify the username and password or configure Single Sign-On.
Ensure the send and receive pipelines are using XmlTransmit and XMLReceive respectively.

And that is it. I'd also point out that you will need to have the WCF LOB Adapter pack installed and the WCF-OracleDB Adapter created in BizTalk Administration Console.

A common error you may encounter is something like this:

Error Description: Microsoft.ServiceModel.Channels.Common.MetadataException: Argument <BtsActionMapping xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">

</BtsActionMapping> is invalid.

The reason for this is because the generated binding file created the BtsActionMapping with a default operation name (in this case GET_AGGREGATE_VALUES). But this operation name has to match the operation name used by the logical send port in the orchestration.

In BizTalk Administration Console, go back to the send port's transport properties and modify the SOAP Action header. Change the operation name to match the logical send port's operation name in your calling orchestration:

</BtsActionMapping>

To get more visibility into communications with Oracle, you can enable listeners to log the WCF-Oracle communications by adding this to the config file:

...

</listeners>

</source>

10 January 2011

Sending Large Files via BizTalk to WCF Service

Recently I was doing some data load tests on my orchestration, which transforms the data and calls a WCF service. I was given over 200 data files and dumped them in to the receive folder. The majority of them worked but a handful of them didn't and I noticed the offending files had much bigger file sizes. The event log error on the BizTalk server looked like this:

Microsoft.XLANGs.Core.XlangSoapException: An error occurred while processing the message, refer to the details section for more information
Message ID: {8E01C976-5029-4C9C-A11C-BF37E3DED66F}
Instance ID: {B573495A-A1BD-49E0-B84B-00DE3EB6AB04}
Error Description: System.ServiceModel.CommunicationException: An error occurred while receiving the HTTP response to http://localhost:3500/DataService.svc/DataService.
This could be due to the service endpoint binding not using the HTTP protocol.
This could also be due to an HTTP request context being aborted by the server (possibly due to the service shutting down).
See server logs for more details.
---> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a receive.
---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.
---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
...

This seemed to indicate an issue on the web server hosting the WCF services. So I enabled tracing in the web.config:


<system.diagnostics>
   <sources>
      <source name="System.ServiceModel" switchValue="Warning">
         <listeners>
            <add name="WCFTrace" />
         </listeners>
      </source>
   </sources>

   <sharedListeners>
      <add name="WCFTrace" type="System.Diagnostics.TextWriterTraceListener" initializeData="C:/Logs/Trace.txt" />
   </sharedListeners>
</system.diagnostics>

and tried again and the trace log wrote this:

<TraceRecord xmlns="http://schemas.microsoft.com/2004/10/E2ETraceEvent/TraceRecord" Severity="Error">
<TraceIdentifier>http://msdn.microsoft.com/en-AU/library/System.ServiceModel.Diagnostics.TraceHandledException.aspx</TraceIdentifier>
<Description>Handling an exception.</Description>
<AppDomain>fbd928cd-22-129391837330249458</AppDomain>
<Exception>
<ExceptionType>System.ServiceModel.CommunicationException, System.ServiceModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType>
<Message>Maximum request length exceeded.</Message>
<StackTrace>at System.ServiceModel.Activation.HostedHttpRequestAsyncResult.GetInputStream()
...

Some Googling of this error suggests to modify the WCF service's web.config file to include this:


<system.web>
  <httpRuntime maxRequestLength="2147483647"/>

So I tried again. Still didn't work but the trace log had a different error:

<TraceRecord xmlns="http://schemas.microsoft.com/2004/10/E2ETraceEvent/TraceRecord" Severity="Error">
<TraceIdentifier>http://msdn.microsoft.com/en-AU/library/System.ServiceModel.Diagnostics.ThrowingException.aspx</TraceIdentifier>
<Description>Throwing an exception.</Description>
<AppDomain>fbd928cd-27-129391852361919458</AppDomain>
<Exception>
<ExceptionType>System.ServiceModel.ProtocolException, System.ServiceModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType>
<Message>The maximum message size quota for incoming messages (65536) has been exceeded. To increase the quota, use the MaxReceivedMessageSize property on the appropriate binding element.</Message>
<StackTrace>at System.ServiceModel.Channels.HttpInput.ThrowHttpProtocolException(String message, HttpStatusCode statusCode, String statusDescription)
...

So following the error message's suggestion, I modified the WCF service's web.config file to specify the binding's maxReceivedMessageSize parameter (the WCF service I am calling is using the basicHttpBinding):


<bindings>
  <basicHttpBinding>
    <binding maxReceivedMessageSize="2147483647">
...

So again I tried, and again another error in the WCF trace log:

<TraceRecord xmlns="http://schemas.microsoft.com/2004/10/E2ETraceEvent/TraceRecord" Severity="Error">
<TraceIdentifier>http://msdn.microsoft.com/en-AU/library/System.ServiceModel.Diagnostics.ThrowingException.aspx</TraceIdentifier>
<Description>Throwing an exception.</Description>
<AppDomain>fbd928cd-24-129391843506509458</AppDomain>
<Exception>
<ExceptionType>System.ServiceModel.Dispatcher.NetDispatcherFaultException, System.ServiceModel, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType>
<Message>The formatter threw an exception while trying to deserialize the message: There was an error while trying to deserialize parameter http://xxx. The InnerException message was 'Maximum number of items that can be serialized or deserialized in an object graph is '65536'. Change the object graph or increase the MaxItemsInObjectGraph quota. '. Please see InnerException for more details.</Message>
<StackTrace>at System.ServiceModel.Dispatcher.DataContractSerializerOperationFormatter.DeserializeParameterPart(XmlDictionaryReader reader, PartInfo part, Boolean isRequest)
...

So again I modified the WCF service's web.config file to fix this issue:


<system.serviceModel>
    ...
    <behaviors>
      <serviceBehaviors>
        <behavior>
           <dataContractSerializer maxItemsInObjectGraph="2147483647"/>
...

and voila, the WCF service is invoked and BizTalk receives the expected response back. Hope this saves you some time.

05 December 2010

Mocking BizTalk XLANGMessage Objects

I have a method that checks for a particular node in an XLANGPart message:

bool IsValidReport(XLANGMessage report){
   XLANGMessage message = report as XLANGMessage;

   if (message != null)
   {
       string status = message[0].GetXPathValue("//*[local-name()='ReportStatus']");

       if (status.Equals("COMPLETE"))
       {
           return true;
       }
   }

   return false;
}

This method is called directly from an orchestration. Note that it relies on reading the value of a node called "ReportStatus" inside the first XLANGPart object.

To unit test this method, I mocked the XLANGMessage and XLANGPart objects:

using Microsoft.XLANGs.BaseTypes;
using Moq;
...

Mock<XLANGMessage> xlangMsgMock = new Mock<XLANGMessage>();
Mock<XLANGPart> xlangPartMock = new Mock<XLANGPart>();

xlangPartMock.Setup(p => p.GetXPathValue(It.IsAny<string>())).Returns("COMPLETE");
xlangMsgMock.SetupGet<XLANGPart>(m => m[0]).Returns(xlangPartMock.Object);

bool isValid = ReportManager.IsValidReport(xlangMsgMock.Object);

xlangPartMock.VerifyAll();
xlangMsgMock.VerifyAll();

Here I've mocked the GetXPathValue method to return "COMPLETE" which will make the IsValidReport method return true. I've then attached this XLANGPart mock to the XLANGMessage mock and called the method to test.

Additional unit tests can be mocked to return other status values to simulate other conditions.

01 December 2010

Oracle Data Access in 32 and 64 bit Environments

I've spent the last couple of days struggling with getting my Oracle data access code (which works well on my 32 bit development machine) to play nice in a 64 bit Windows Server 2008 R2 environment. In the end the solution was very straightforward and the code can be compiled using the 32 bit version of the Oracle.DataAccess.dll and configured to run in a 64 bit environment without having to rebuild by editing the configuration file below:

<system.data>
 <DbProviderFactories>
    <!-- this is the 64 bit version of the Oracle.DataAccess.dll -->
    <add name="Oracle Data Provider for .NET" invariant="Oracle.DataAccess.Client"
    description="Oracle Data Provider for .NET"
    type="Oracle.DataAccess.Client.OracleClientFactory, Oracle.DataAccess, Version=2.111.7.0,
    Culture=neutral, PublicKeyToken=89b483f429c47342" />
 </DbProviderFactories>
</system.data>

<runtime>
 <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
 <!-- redirect all requests for Oracle.DataAccess.dll to the 64 bit version -->
    <dependentAssembly>
       <assemblyIdentity name="Oracle.DataAccess" publicKeyToken="89b483f429c47342" culture="neutral" />
       <bindingRedirect oldVersion="0.0.0.0-100.100.100.100" newVersion="2.111.7.0" />
    </dependentAssembly>
 </assemblyBinding>
</runtime>

As usual, ensure that the right assemblies are in the GAC.

UPDATE: my colleague was experiencing some unusual behaviour in his development environment and after some investigation, it turns out he installed the wrong Oracle components. If the .NET Oracle component is all you require, then when you are running the Oracle installer, choose the custom option and select the Oracle Data Provider for .NET 2.0 11.x.x.x only. This will install the redirection assemblies into the GAC and the above solution will work.

24 November 2010

Mapping a Grouped Flat File

Recently I had to process a type of flat file with a rather tricky format that looked something like this (simplified for clarity):

H,INV_201008.CSV,2010-09-01

G,1011,CODE_001

D,23562,100.00,32,2010-05-05

D,73522,110.00,33,2010-05-06

D,82643,104.00,11,2010-05-07

G,1012,CODE_002

D,23553,99.00,32,2010-05-05

D,73463,156.60,11,2010-05-06

D,96473,122.10,88,2010-05-07

G,1013,CODE_003

D,25435,30.00,14,2010-05-05

D,13435,80.00,8,2010-05-06

D,97563,42.00,22,2010-05-07

A data file consists of one header (H) row, multiple group (G) rows and multiple data (D) rows for each G row.

After using BizTalk's flat file schema wizard, I had to make some manual modifications to the schema to accommodate the unbounded repeating set of G and D rows by treating G as a record,

D as a repeating record and then wrapping the G and D elements within an <xs:sequence maxOccurs="unbounded"> tag.

BizTalk will generate an xml representation of the flat file that looks like this:

</Header>

</GroupRow>

</DataRow>

</DataRow>

</DataRow>

</GroupRow>

</DataRow>

</DataRow>

</DataRow>

</GroupRow>

</DataRow>

</DataRow>

</DataRow>

As can be seen, when BizTalk disassembles the flat file, the xml data is in a flat structure. The challenge is to transform it to a hierarchical structure like this:

</Header>

<Group>

<Data>

</Data>

<Data>

</Data>

<Data>

</Data>

</Group>

<Group>

<Data>

</Data>

<Data>

</Data>

<Data>

</Data>

</Group>

<Group>

<Data>

</Data>

<Data>

</Data>

<Data>

</Data>

</Group>

I tried several approaches but couldn't seem to get BizTalk's mapper to generate what I wanted. I then tried the custom XSL route and after a lot of experimentation, I got it working.

Creating the <Group> nodes was the easy bit (using an <xsl:for-each> and selecting the <GroupRow> nodes). Creating the filter expression for <Data> nodes within each group node iteration was the hardest part. In a nutshell, the <xsl:for-each> select statement was:

following-sibling::*[local-name()='DataRow' and (count(preceding-sibling::*[local-name()='GroupRow'])) = $marker]

where $marker is a variable that I set at the start of each group node iteration like this:

<xsl:variable name="marker" select="count(preceding-sibling::*[local-name()='GroupRow'])+1" />

Hope this tip will save someone some time.

23 November 2010

BizTalk and X509 Certificate Permissions

Our client needs to retrieve data from a third party web service over SSL using an X509 certificate via BizTalk. The setup is to use a wsdl generated C# proxy class, subclass it and configure the certificate in the constructor:

//Note: ClientCertificates is a property of the base class System.Web.Services.Protocols.SoapHttpClientProtocol
this.ClientCertificates.Add(new System.Security.Cryptography.X509Certificates.X509Certificate(filePath, password));

Then in BizTalk Server Administration Console, the send port is using a SOAP adapter configured to use the proxy class and method by specifying the assembly.

One thing to note is that although the code above is configured to read the certificate's file path and password as app settings from the BTSNTSvc.exe.config file (BTSNTSvc64.exe.config if the handler is running in 64 bit mode), the certificate still needs to be imported into the machine's certificate store with the right permissions. I found this out the long way when things were running smoothly on my development machine but kept failing on the integration server with this message:

A message sent to adapter "SOAP" on send port "xxx" with URI "https://..." is suspended.
Error details: WebException: The request failed with HTTP status 403 Forbidden.

Not a very helpful message. Going on a hunch, I decided to trace the network activity using a very simple yet powerful .NET tracing technique I found at http://blogs.msdn.com/b/dgorti/archive/2005/09/18/471003.aspx. I reconfigured the BizTalk service to start logging all network activities and found this in the logs:

System.Net Information: 0 : [1720] SecureChannel#65709741 - Cannot find the certificate in either the LocalMachine store or the CurrentUser store.

I checked that the certificate was indeed in the certificate store. After a bit more Googling, the culprit turned out to be permissions. To fix the issue:

In mmc, right click on the certificate and choose All Tasks > Manage Private Keys...
Click on the Add... button and choose the BizTalk Application Users group (or whichever group that contains the account that runs your BizTalk host instance).
Ensure this group has Full control permissions then click OK.

Once that was done, my SOAP requests were working as expected.

08 August 2015

BizTalk Certification

05 May 2011

Oracle Adapters and BizTalk

10 January 2011

Sending Large Files via BizTalk to WCF Service

05 December 2010

Mocking BizTalk XLANGMessage Objects

01 December 2010

Oracle Data Access in 32 and 64 bit Environments

24 November 2010

Mapping a Grouped Flat File

23 November 2010

BizTalk and X509 Certificate Permissions

Blog Archive