Simon on Software

I’ve recently worked on some ETL tasks where there is a requirement to to show what has changed between different versions of datasets. LINQ and a custom IEqualityComparer eats this problem for breakfast:

WCF defaults to buffering the entire message before transmitting. It is possible to stream your data so data can start to be processed on the server or client before it entirely transmitted.

Comprehensive documentation can be found on

Set server app.config to be:

<bindings>
   <netTcpBinding>
      <binding name="customNetTcpBinding"
               maxBufferSize="200000000"
               maxReceivedMessageSize="200000000"
               transferMode="Streamed">
      </binding>
    </netTcpBinding>
</bindings>

The above should allow files of up to about 190mb to be streamed.

The key attribute is:

 transferMode="Streamed" - default is buffered

The client config required is.

<bindings>
   <netTcpBinding>
      <binding name="NetTcpBinding_IFileUpload" transferMode="Streamed" />
   </netTcpBinding&gt;
</bindings>

For a complete example, our response type definition:

public class UploadFileResponse
{
   public string Message { get; set; }
}

Service interface:

[ServiceContract]
public interface IFileUpload
{
   [OperationContract]
   UploadFileResponse UploadFileViaStream(Stream fileStream);
}

Service class implementation:

public class FileUploadService : IFileUpload
{
   public UploadFileResponse UploadFileViaStream(Stream stream)
   {
      var response = new UploadFileResponse();

      using (var fileStream = File.Create("teststream"))
      {
         stream.CopyTo(fileStream);
      }

      response.Message = "Successfully completed";
      return response;
   }
}

And if we look at Microsoft Service Trace Viewer we can see that it is indeed streaming our data:

<E2ETraceEvent xmlns="http://schemas.microsoft.com/2004/06/E2ETraceEvent">
	<System xmlns="http://schemas.microsoft.com/2004/06/windows/eventlog/system">
		<EventID>0</EventID>
			<Type>3</Type>
			<SubType Name="Information">0</SubType>
			<Level>8</Level>
			<TimeCreated SystemTime="2014-03-06T18:17:25.8188503Z" />
			<Source Name="System.ServiceModel.MessageLogging" />
			<Correlation ActivityID="{00000000-0000-0000-0000-000000000000}" />
			<Execution ProcessName="WCFLargeFileAPI.vshost" ProcessID="2984" ThreadID="11" />
			<Channel />
	</System>
<ApplicationData>
<TraceData>
	<DataItem>
		<MessageLogTraceRecord Time="2014-03-06T18:17:25.8188503+00:00" Source="ServiceLevelReceiveRequest" Type="System.ServiceModel.Channels.StreamedMessage" xmlns="http://schemas.microsoft.com/2004/06/ServiceModel/Management/MessageTrace">
		<s:Envelope xmlns:s="http://www.w3.org/2003/05/soap-envelope" xmlns:a="http://www.w3.org/2005/08/addressing">
			<s:Header>
				<a:Action s:mustUnderstand="1">http://tempuri.org/IFileUpload/UploadFileViaStream</a:Action>
				<a:MessageID>urn:uuid:7e9efd61-3868-481c-ab5d-17c19f93526a</a:MessageID>
				<a:ReplyTo>
				<a:Address>http://www.w3.org/2005/08/addressing/anonymous</a:Address>
			</a:ReplyTo>
			<VsDebuggerCausalityData xmlns="http://schemas.microsoft.com/vstudio/diagnostics/servicemodelsink"<uIDPo1d4u3tgmU9MjjApxUoHeD8AAAAABU27dsajsEGNM7x3QCkSz7rxDZ3MpsRFrpqvfafNFOAACQAA</VsDebuggerCausalityData>
			<a:To s:mustUnderstand="1">net.tcp://localhost:7891/<a:To>
			</s:Header>
		<s:Body>... stream ...</s:Body>
		</s:Envelope>
		</MessageLogTraceRecord>
	</DataItem>
</TraceData>
</ApplicationData>
</E2ETraceEvent>

Create a host application, such as a Console Application for easy testing

Add a reference to System.ServiceModel

Add a service interface such as :

 
[ServiceContract]     
public interface IHelloWorldService     
{         
    [OperationContract]         
    string SayHello(string to);     
} 

Add a service implementation :

public class HelloWorldService : IHelloWorldService
{
    public string SayHello(string to)
    {
       return string.Format("Hello, {0}!", to);
    }
}

Host the service as you wish:

class Program
{
    static void Main(string[] args)
    {
        var host = new ServiceHost(typeof(HelloWorldService));
        host.Open();
        Console.WriteLine("Press any key to close");
            Console.ReadKey(true);
            host.Close();
        }
    }

And a minimal app.config file where all the magic happens:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
    <startup> 
        <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5" />
    </startup>
  <system.serviceModel>
    <behaviors>
      <serviceBehaviors>
        <behavior name="MexBehaviour">
          <serviceMetadata httpGetEnabled="false" />
        </behavior>
      </serviceBehaviors>
    </behaviors>
    <services>
      <service behaviorConfiguration="MexBehaviour" name="Server.HelloWorldService">
        <endpoint address="" binding="netTcpBinding"
            contract="Server.IHelloWorldService" />
        <endpoint address="mex" binding="mexTcpBinding" bindingConfiguration=""
            contract="IMetadataExchange" />
        <host>
          <baseAddresses>
            <add baseAddress="net.tcp://localhost:7891" />
          </baseAddresses>
        </host>
      </service>
    </services>
  </system.serviceModel>
</configuration>

And it works:

Read all about it

More for my own recall than anything, but this feature of Moq that I hadn’t seen before solved an issue. If you need to return different results on each invocation of a method:

Moq SetupSequence

I’m a complete CouchDB novice. Here is what I learnt in the first few hours.

What is it?

tldr – It’s a document database, that heavily uses http for querying and administration

Installation

Setting up a dev environment on a mac is pretty easy. I have homebrew installed as my package manager.

brew search couch

showed that the package is called ‘couchdb’

brew install couchdb

went off and did it’s thing. After 45 minutes or so, it seemed to have hung on the make task. The CouchDB Mac installation instructions suggested using the command

brew install -v couchdb

and this worked for me.

Couch is now installed in /usr/local/bin/couchdb on my system.

We start the server with couchdb and see the following:

Apache CouchDB 1.4.0 (LogLevel=info) is starting.
Apache CouchDB has started. Time to relax.
[info] [] Apache CouchDB has started on http://127.0.0.1:5984/

In another window, we can act as a client – curl http://127.0.0.1:5984/ which tells us

{"couchdb":"Welcome","uuid":"8e9e41844b1be7e68475d288b5d14bc3","version":"1.4.0","vendor":{"version":"1.4.0-1","name":"Homebrew"}}

Web Admin via Futon

Futon is a web interface to CouchDB. We can perform database administration tasks, and create and edit documents. I was able to create a new database and insert a simple document.

Note that CouchDB has assigned and Id and a Revision to the document. You can assign your own unique identifiers should you wish.
You can even ask CouchDB to give you a range of UUIDs :

curl -X GET http://127.0.0.1:5984/_uuids?count=5

{"uuids":"0b225462ddccbb8f19ec612ef70319f9","0b225462ddccbb8f19ec612ef7031eba",
"0b225462ddccbb8f19ec612ef7032269","0b225462ddccbb8f19ec612ef7032e8d",
"0b225462ddccbb8f19ec612ef7032f8d"]}

Please don’t use the above UUIDs, they belong to me

Retrieving data with node.js

Following the code from https://github.com/cloudhead/cradle it was super simple to query the document we have already created.
After we have run npm install cradle the following was enough to get us started.

var cradle = require('cradle');
var db = new(cradle.Connection)().database('hello_couch');
db.get('0b225462ddccbb8f19ec612ef70007fd', function(err, doc) {
  console.log(doc);
});

which returns

{ _id: '0b225462ddccbb8f19ec612ef70007fd',
       _rev: '1-7f6fc4765d88ba5da29b5f5938b46c2c',
       message: 'hello from couch db' }

Getting started was simple enough. Now to learn more about the ecosystem and how to use it in a real application.

An extension of my short term memory:

I’ve been taking a look at emacs – this is my brain dump so far – mainly taken from the in-built tutorial

Many moons ago (at least 18 months ago), I had a stored procedure that let me search for text in DML in the databases. Yeah, so I could search for “Order” and it would find all mentions in SPs, functiosn etc.

Here is my new version:

SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED 

SELECT 
        SUBSTRING (qt.text,(qs.statement_start_offset/2) + 1, 
        ((CASE WHEN qs.statement_end_offset = -1 
        THEN LEN(CONVERT(NVARCHAR(MAX), qt.text)) * 2 
        ELSE qs.statement_end_offset 
        END - qs.statement_start_offset)/2) + 1) AS [Individual Query] 
, qt.text AS [Parent Query] 
, DB_NAME(qt.dbid) AS DatabaseName 
, qp.query_plan 
FROM sys.dm_exec_query_stats qs 
CROSS APPLY sys.dm_exec_sql_text(qs.sql_handle) as qt 
CROSS APPLY sys.dm_exec_query_plan(qs.plan_handle) qp 
WHERE SUBSTRING (qt.text,(qs.statement_start_offset/2) + 1, 
        ((CASE WHEN qs.statement_end_offset = -1 
        THEN LEN(CONVERT(NVARCHAR(MAX), qt.text)) * 2 
        ELSE qs.statement_end_offset 
        END - qs.statement_start_offset)/2) + 1) 
LIKE '@searchtext' 
and DB_NAME(qt.dbid) = @database