Introducing resilience with polly

So now we have the services talking to one another, we should think about building in some resilience – essentially a way for the apps to retry a connect in case of a transient fault, e.g. a network blip.

We’ll revert back to using the Typed Http Client Factories at Stage 3 of the previous post. As such the source code should have the following:

Startup.cs

services.AddHttpClient();

StudentController.cs

public class StudentController : Controller
{
    private readonly StudentRecordSystemService _studentRecordService;
    public StudentController(StudentRecordSystemService studentRecordService)
    {
         _studentRecordService = studentRecordService;
    }

    public async Task Index()
    {
        List students = await _studentRecordService.GetStudentsAsync();
        return View(students);
    }
}

Services/StudentRecordSystemService.cs

public class StudentRecordSystemService
{
     public HttpClient Client { get; }
     public StudentRecordSystemService(HttpClient client)
     {
         client.BaseAddress = new Uri("<a>http://localhost:5100/");</a>
         // Optionally add additional header information, e.g. keys etc... not required in this example.
         Client = client;
     }

    public async Task&lt;List&gt; GetStudentsAsync()
    {
         var response = await Client.GetAsync("api/Student");         
         response.EnsureSuccessStatusCode();
         return await response.Content.ReadAsAsync&lt;List&gt;();
     }
}

Once again, start up both services and check that the expected data is displayed.

Introducing Polly

1. In the ViewerWebApplication, right click on the Dependencies and ‘Manage NuGet Packages…’. Then search for Polly.Extensions.Http and click Install.

2. Update StudentRecordSystemService.cs to match the following (note the introduction of the logger via DI):

private readonly ILogger _logger;

public StudentRecordSystemService(HttpClient client, ILogger logger)

{
     _logger = logger;
     client.BaseAddress = new Uri("<a>http://localhost:5100/");</a>
     // Optionally add additional header information, e.g. keys etc... not required in this example.
     Client = client;
}

public async Task&lt;List&gt; GetStudentsAsync()
{
     var response = await Policy
             .Handle()
             .WaitAndRetryAsync(3, retryAttempt =&gt; TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)), (exception, timeSpan, context) =&gt;
             {
                 _logger.LogError($"Failed. Waiting {timeSpan}. Exception: {exception.Message}.");
             })
             .ExecuteAsync(async () =&gt; await Client.GetAsync("api/Student"));
    response.EnsureSuccessStatusCode();
    return await response.Content.ReadAsAsync&lt;List&gt;();
}

And the Startup.cs has also updated to include the LoggerFactory:

public Startup(IConfiguration configuration, ILoggerFactory loggerFactory)
{
     Configuration = configuration;
     LoggerFactory = loggerFactory;
}

public IConfiguration Configuration { get; }
public ILoggerFactory LoggerFactory { get; }

Now if you start the DataViewer app, but not the StudentRecordSystemService, the page will now attempt multiple times (with an increasing time delay between calls, and logging between each event) before eventually giving up and throwing an exception.

Ultimately we’d want to declare these kind of Polly policies just once, and thats something we’ll look at next.

Advertisements

Communicating Between the Apps – HttpClient, HttpClientFactory, Refit

(Apologies – this turned into quite a lengthy blog post. Hopefully it took longer to write than it’ll take to follow along – if you need to, skip to the final bits about Typed HttpClientFactories and Refit).

(Apologies 2 – turned out I pasted the wrong code in for the Typed Http Clients – now amended. Thanks for the feedback).

To keep things fairly simple, the main aim of this blog post is to retrieve some data from the Student Record System (SRS) Web API application by a call from the DataViewer Web Application and display the data received. Not much love will go into the display of the data (that may come in a later blog post), this will be focusing on the concepts of communicating.

There are a number of ways to achieve this in .NET Core – some that are suitable for production code, and some, well, less so. I’m going to try and walk through each method in turn so we can explore them all, and perhaps try and point out some of the limitations with each approach (and defer to related blog articles from authors who can write much better than myself and cover the details in a much greater depth).

First Step – Removing SSL

To avoid having to set-up SSL certificates, change the launchSettings for the SRS project to be http – to do this expand the Properties node and open the launchSettings.json file. Change the address from the previous blog post from https://localhost:5100 to http://localhost:5100.

Setting Up Some Data

In the Web API application (SRS), click on the Project and click Add->New Folder and call the folder Models.
Then right click on this new folder and add a class and name it ‘Student’.

We’ll then add some properties to this Student class, keeping them simple for now.

(Quick Tip: A short-cut in Visual Studio exists to type prop and then press Tab and Tab afterwards. This generates the scaffold code for a property.)

We’re going to add strings for First Name, Last Name, Email and Student ID.

(Quick Tip: It would also be nice to include a photo, but we’ll flag this as something to come back to. You can flag TODO items in the code by using the comment syntax with TODO – as shown in the code below. All these entries then show up in the Task List in Visual Studio.)

We also need to add 2 constructors – one empty to aid serialisation (something we’ll come back to later) and then another to all the creation of a new student with the properties being set at creation.

(Quick Tip: A short-cut in Visual Studio exists to type ctor[tab][tab] and this will generate to constructor signature for you.)

namespace StudentRecordSystem.Models
{
    public class Student
    {
        // TODO: Add Student Photo/Image
        public string FirstName { get; set; }
        public string LastName { get; set; }
        public string Email { get; set; }
        public string StudentId { get; set; }
    }


        public Student()
        {
        }


        public Student(string firstName, string lastName, string email, string studentId)
        {
            FirstName = firstName;
            LastName = lastName;
            Email = email;
            StudentId = studentId;
        }
}

Next we’ll introduce a controller to then serve up a list of students. On the controllers folder, right click and Add->Controller…
Select API – Empty Controller and enter StudentController as the name.

In this controller we need to first annotate the controller itself with [Produces(“application/json”)] – I think this is the default anyway if not specified, but best to be explicit.

Then implement the Get method to return some student data – see below for the example code:

    [Route("api/[controller]")]
    [ApiController]
    [Produces("application/json")]
    public class StudentController : ControllerBase
    {
        [HttpGet]
        public IActionResult Get()
        {
            List students = new List();
            students.Add(new Student("Anna", "Annabel", "a.a@uni.ac.uk", "1234"));
            students.Add(new Student("Brian", "Beater", "b.b@uni.ac.uk", "1235"));
            return Ok(students);
        }
    }

With this in place, start the app again (as per the previous blog post) and then view the /api/Student URL and you should get JSON output of the students specified above.

With this in place now, we can return to the Web App and make a call to this API and start to display the data.

Multiple Ways To Call Web APIs

1. The HttpClient – The simplest, but perhaps problematic

In the DataViewer Web App, right click on the Controllers folder and ‘Add’ and New Controller. This time select Empty MVC controller, and call it StudentController.

Within this controller will be default method called Index(). Within it place the following code:

public async Task Index()
{
var url = “http://localhost:5100/api/Student&#8221;;
var client = new HttpClient();
var response = await client.GetStringAsync(url);
return response;
//return View();
}

The code above essentially defines the URL to call, and then creates an HttpClient object. It then uses the helper method .GetStringAsync to call the address and read the string returned. Finally we return this string straight to the browser. In a future step we’ll look at tidying up this display.

Thats it, but this method comes with a caveat. Steve Gordon covers this in excellent detail in his little series on HttpClientFactory, so I don’t plan to recover it here ( https://www.stevejgordon.co.uk/introduction-to-httpclientfactory-aspnetcore) (and I’ve also linked to the ASP.NET Monsters post about this issues), but essentially every time this method is executed we create a new HttpClient, and secondly the lifetime of the client isn’t handled very well, which could lead to issues in a scaled environment.

1b. Formatting the data output

Before going too much further, lets take the data returned and format it. For this we’ll use a 3rd party library – Newtonsoft.Json.

a. Right click on the DataViewer application project and select Manage NuGet Packages…
b. Ensure the ‘Browse’ tab is selected. You may not need to search for anything, it was the first list library for me. (otherwise search for Newtonsoft). Click and select the ‘Install’ option. (and OK any other pop-ups).
c. Next we need to add a model to represent the data returned. This will essentially be the same as the Student class created in the SRS project. As such, on the Models folder, right click and add new class, named Student. Enter the following code:

    public class Student
    {
        // TODO: Add Student Photo/Image
        public string FirstName { get; set; }
        public string LastName { get; set; }
        public string Email { get; set; }
        public string StudentId { get; set; }


        public Student()
        {
        }


        public Student(string firstName, string lastName, string email, string studentId)
        {
            FirstName = firstName;
            LastName = lastName;
            Email = email;
            StudentId = studentId;
        }
    }

d. Next, update our controller to now convert the Json String to a List of Student Models. The code is as follows:

        public async Task Index()
        {
            var url = "http://localhost:5100/api/Student";
            var client = new HttpClient();
            var response = await client.GetStringAsync(url);
            List students = JsonConvert.DeserializeObject(response);
            return View(students);
        }

The code above is very similar to the code previously provided except:
i. We’ve changed the return type.
ii. We’ve introduced JsonConvert (from Newtonsoft.Json) to convert the string returned to a List of students.
iii. We return to a view (which is created next) which uses the list of students to create a table (nothing pretty yet, but we’ll tidy up to use Bootstrap in a future blog post).

e. Create a new View. First, expand the Views folder and then create a new folder of ‘Student’ (NB. it must match the Controller name provided previously). Then right click on this new folder and create a View, give it the name of Index and create it. Enter the following code to just output a table:

@model List
@{
    ViewData["Title"] = "Index";
}


<h1>Index</h1>
<table>
    <tr>
        <th>Photo</th>
        <th>First Name</th>
        <th>Last Name</th>
        <th>Email</th>
        <th>Student ID</th>
    </tr>
    @{
        foreach (var item in @Model)
        {
            <tr>
                <td>TBD</td>
                <td>@item.FirstName</td>
                <td>@item.LastName</td>
                <td>@item.Email</td>
                <td>@item.StudentId</td>
            </tr>
        }
    }
</table>

Run up both apps, and now see that you’re getting a Web Page with a table of the student data, as opposed to the raw JSON.

2. Using Named HttpClientFactories

An alternative method by using HttpClientFactories was introduced with .NET Core 2.1. In this approach you can declare a factory to be used instead of creating the client upon every request. Some of the benefits are slight efficiencies, the factory will manage the lifetimes of the HttpClients on your behalf and you’ve a central place for all requests to a location in your app, rather than potentially being scattered throughout your application (a good approach should you ever need to change the location of the other service). Its probably best explained with an example.

In Startup.cs of the DataViewerApp, add the following code to the ConfigureServices() method:

            services.AddHttpClient("StudentRecordSystem", s =>
            {
                s.BaseAddress = new Uri("http://localhost:5100/");
                // Optionally add additional header information, e.g. keys etc... not required in this example.
            });

Next we need to use Dependency Injection to pass the factory to the controller, before then using it instead of the calls we were previously making.

So in the StudentController.cs, introduce a Constructor (Remember the short-hand tip from earlier). Then make the controller look as follows:

    public class StudentController : Controller
    {
        private readonly IHttpClientFactory _clientFactory;

        public StudentController(IHttpClientFactory clientFactory)
        {
            _clientFactory = clientFactory;
        }

        public async Task Index()
        {
            List students = new List();
            var request = new HttpRequestMessage(HttpMethod.Get, "api/Student");
            var client = _clientFactory.CreateClient("StudentRecordSystem");

            var response = await client.SendAsync(request);

            if (response.IsSuccessStatusCode)
            {
                students = await response.Content.ReadAsAsync();
            }
            return View(students);
        }
    }

Run up the apps and ultimately nothing has changed, but we’ve made the code a little bit better along the way.

3. Using Typed HttpClientFactories

One major drawback of the ‘Named’ approach above is that the strings need to match in both places and as such is liable to typo errors (and the compiler won’t help you with this – you’ll get run time problems). Also the calls to specific endpoints are also still scattered throughout the code.

An alternative method to overcome some of these limitations is typed clients. Again, an example may be the easiest way to explain.

In the DataViewer app, right click on the project and Add->New Folder and call it ‘Services’. Then right click on the folder and Add->Class… and call this StudentRecordSystemService. Then implement the code outlined below:

    public class StudentRecordSystemService
    {
        public HttpClient Client { get; }

        public StudentRecordSystemService(HttpClient client)
        {
            client.BaseAddress = new Uri("http://localhost:5100/");
            // Optionally add additional header information, e.g. keys etc... not required in this example.
            Client = client;
        }

        public async Task GetStudentsAsync()
        {
            var response = await Client.GetAsync("api/Student");
            response.EnsureSuccessStatusCode();
            return await response.Content.ReadAsAsync();
        }
    }

This seems a bit more involved, but ultimately derives a number of benefits in the calling code base – the changes which are outlined below. In Startup.cs, replace the code we added previously with the following:

services.AddHttpClient();

And then change the code in the StudentController to match the code below:

    public class StudentController : Controller
    {
        private readonly StudentRecordSystemService _studentRecordService;

        public StudentController(StudentRecordSystemService studentRecordService)
        {
            _studentRecordService = studentRecordService;
        }

        public async Task Index()
        {
            List students = await _studentRecordService.GetStudentsAsync();
            return View(students);
        }
    }

Again, start the apps and check our student table comes back as expected. We’ve now got the benefit of all the calls being made in a single place, using a typed client factory, alongside creating a folder for a common area for all API calls to be stored and sit side-by-side. Finally, the code in the callers becomes much more simplified.

4. Using Refit

Finally, we can refine things further by using a 3rd party library called refit.

First, add a reference to the NuGet package ‘Refit’ (right click on DataViewer Project and click Manage NuGet Packages). Then browse for the ‘Refit’ library and install it.

Then add an Interface (Add->New Class) to the Services folder and implement the following code:

    public interface IStudentRecordSystemService
    {
        [Get("/api/Student")]
        Task GetStudentAsync();
    }

Next change the StudentController to match the following:

        public async Task Index()
        {
            // Missing try...catch for error handling.
            var srsClient = RestService.For("http://localhost:5100/");
            List students = await srsClient.GetStudentAsync();
            return View(students);
        }

And thats it. We can now remove code in the Startup.cs if we wish, along with the StudentRecordSystemService.cs. Again, run the app and ensure everything is running.

References:
https://docs.microsoft.com/en-us/aspnet/core/fundamentals/http-requests?view=aspnetcore-2.2

HttpClientFactory in ASP.NET Core 2.1 (Part 1)

https://aspnetmonsters.com/2016/08/2016-08-27-httpclientwrong/

https://www.projectcodify.com/creating-a-rest-api-wrapper-using-refit

In this 1st article I’m going to use a lot of scaffolding code from Microsoft to get up and running very quickly. I’m going to create a Web Site application (based upon MVC) and a separate Web API Application. I’m going to keep them in a single solution for now just to even further simplify the learning curve. They may break out at a late stage once we start to consider automated testing techniques.

I’m going to use Visual Studio (and 2019 version in particular) and use .NET Core 2.2. I’ll hopefully migrate to .NET Core 3.0 as soon as possible after release. However there is no reason why these can’t be created using the .NET Core command line tools and I’ll link to relevant articles showing how to do this at the end of this post. I’m also not aware of anything here that is specifically .NET Core 2.2 specific, so things may be okay with earlier versions (but please let me know if you find anything and I’ll amend).

The Web Application

At this stage I’m not sure what role this Web Application is going to fulfill in the ecosystem so I intend to use the very scaffold default MVC website project from Microsoft. This should allow flexibility going forward, or alternatively it may get fully replaced in the future. For now it’s sole purpose to display data from the other microservices that will be introduced.

1. Start Visual Studio and select ‘Create New Project’.
2. Select ‘ASP.NET Core Web Application’ and click ‘Next’.

createnewproject1-2

3. Select a suitable name (I’ve got for ViewerWebApplication) and I’ve place it under a folder called UniversityExample.
4. Then select Web Application (Model-View-Controller) and click Create.

createnewproject1-4

Once Visual Studio has created the application, hit F5 (or the Green play button with IIS Express) and check that the website runs. It should look similar to below:

createnewproject1-5

We’ll explore some of the code that has been generated as we start to bring in new functionality later in the series.

The Web API Application

Next. in the same solution, we’re going to add the out-the-box Web API application.

1. Right click on the solution and click ‘Add’ and ‘New Project’.
2. Again select ASP.NET Core Web Application and click Next.
3. For project name, call it ‘StudentRecordSystem’ and click Create.
4. On the next screen click ‘API’ and then click ‘Create’.

When launching applications, Visual Studio will launch them on random ports. In order to fix this, for the Web API application just created, expand ‘Properties’ and click on the ‘launchSettings.json’ file. Change the bottom section to specify a port for the https:// address and remove the http:// entry altogether. Your new file should look similar to the following:

{
  "$schema": "http://json.schemastore.org/launchsettings.json",
  "iisSettings": {
    "windowsAuthentication": false, 
    "anonymousAuthentication": true, 
    "iisExpress": {
      "applicationUrl": "http://localhost:62421",
      "sslPort": 44382
    }
  },
  "profiles": {
    "IIS Express": {
      "commandName": "IISExpress",
      "launchBrowser": true,
      "launchUrl": "api/values",
      "environmentVariables": {
        "ASPNETCORE_ENVIRONMENT": "Development"
      }
    },
    "StudentRecordSystem": {
      "commandName": "Project",
      "launchBrowser": true,
      "launchUrl": "api/values",
      "applicationUrl": "https://localhost:5100",
      "environmentVariables": {
        "ASPNETCORE_ENVIRONMENT": "Development"
      }
    }
  }
}

Finally, rather than using Visual Studio to run this app, we’ll use the command line to it. Right click on the project ‘StudentRecordSystem’ and find the entry ‘Open Folder in File Explorer’. Once Windows Explorer opens, in the address bar, type cmd and press enter. This will launch a command line window at the directory specified.

On the command line type ‘dotnet run’ and wait for some output telling you the app is running, similar to below:

createnewproject2-1

Now head to a browser, and enter the address above, e.g. https://localhost:5001/api/Values and you should get the following output:

createnewproject2-2

Don’t worry about any security certificate warnings just for now.

Nothing too exciting just yet, but next we’ll look to introduce a new API controller which will provide some hardcoded student information, which we’ll then display on the Web Site application, creating a connection between the 2 applications.

The Kitchen sink project

So I’m not quite sure where this little blog series is going to go, but I’m going to try and develop a semi serious distributed .NET Core microservices solution. One thing about the series is it will be the ‘kitchen sink’ project – any technology I read about, hear about or otherwise will be thrown at it, primarily for my own learning purposes but perhaps to aid other readers as well.

I’m proposing to base it on the layout of a University’s architecture, so students and their learning might be a key focus. I’ve chosen this domain as (a) its one I know relatively well and (b) for others it should be relatively simple domain to understand and thus allow focus on the technology.

The below is all subject to change, but a broad outline for the kind of blog posts I intend to publish are below (and I’ll change these to link to the blog posts as they go live). The series will start with a focus on developing the ecosystem and development environment – by getting these in place early this should hopefully lead to faster subsequent development.

  • The Start – A .NET Core Web Site (A viewer of data) and a .NET Core Web API project (The Student Record System as a repository of data?)
  • Using Source Control with Git and Github
  • .NET Core Health Checks and a Health Check Microservice to view status
  • Logging Concepts
    • Serilog
    • ELK Stack exploration
  • Testing
    • Unit Testing with MSUnit
    • Selenium Testing for User Interfaces
  • Containers with Docker on Windows
    • Images and Containers
    • Mutli-stage images – build and run steps
    • Specifying environment variables
    • Mounting volumes & persistent storage
    • Docker compose
  • Continuous Integration & Continuous Deployment
    • TeamCity?
    • Azure?
  • Developing the Student Record System Web API:
    • Data
      • Student Data
      • Module Data
      • Programme Data
      • Associated Student Enrolments
    • Hardcoding the data above
    • Persistent storage to SQL Express with Entity Framework Core
    • Docker and SQL Express?
  • Raising Events from the Student Record System
    • RabbitMQ (via Docker?)
    • Sending events from SRS to RabbitMQ
  • Processing Events in (new) Virtual Learning Environment API Application

Other topics that will be introduced as some point will be things akin to Polly and Bundle Minifer.

Connecting to an oracle database from microsoft .net

Work has changed for me slightly over the past few years, and I’ve moved into the world of Project Management. I’ve however got a chance again now to get my hands ‘dirty’ with some .NET coding – its been a few year since I’ve done any in anger – so I’m trying to gain a refresh into the current platform and its technology. Expect a few .NET 101 posts to follow this one.

We’re looking significantly to adopt .NET Core as we’ve got a great greenfield project on the horizon. However much of our infrastructure is based on Oracle and without a .NET Core supported route, I think we might have to fall back to .NET Framework 4.6.X, at least in the short term.

This blog post is to look at how .NET apps can connect to, specifically, Oracle Databases.

Pre-Requisites

You need to install Oracle Data Access Components (ODAC) with Developer Tools for Visual Studio (from here). Extract the zip, and click setup.exe and follow the defaults.

1. Connecting via a Windows Console App (.NET Framework 4.6.1)

  • Open Visual Studio 2015, and click File->New->Project.
  • Then select Visual C#->Windows and Console Application, confirming that .NET Framework 4.6.1 is selected from the drop-down menu.

    image

  • Once the project has been created, add 2 new packages via NuGet (Tools->NuGet Package Manager->Manage NuGet Packages for Solution…)
    image
  • Click Browse from the top menu bar, and then search from EntityFramework by Microsoft (its likely to be listed near the top anywhere). Select and install.
  • Then search for ODP and install the ‘Oracle.ManagedDataAccess.EntityFramework’ by Oracle.
  • Next, close down NuGet Package Manager and right click on the project and Add New Item…
  • From here select Data and ADO.NET Entity Data Model
    image
  • Then select ‘Code First from Database’
  • To create a new Connection I’ve found adding the details the installed tnsnames.ora file to be the best way – I’m not in a position to share mine, but there should be examples out there on the Internet.
  • Select the appropriate connection, enter the username and password and click Next.
  • I then select all the tables that are available in the database.
  • Click Finish. This will generate a number of files, that should relate primarily to the tables you’ve selected.
  • Then enter the following code into your program (note, the name of the database may alter depending on what you enter in the ADO.NET file – I think it defaults to Model – my example below uses GLJ). The syntax for the query you’re running may also alter depending on your data columns.
    image

Running this provides a list of modules that we have stored in a test database from Oracle.

2. Connecting to Oracle from a .NET Core Application

I’m really not expecting this to work – I’m not aware of any announcement about Oracle and .NET Core support at this time – certainly my Google searches haven’t been that fruitful. The closest I’ve got is this discussion on the OTN. And looking back over the @OracleDotNet twitter feed I can’t see anything having been announced.

Anyway, back to Visual Studio 2015 and this time New->Project. And then Visual C#->.NET Core and Console Application (.NET Core).

Then using NuGet to add the 2 packages as per the previous guide for a standard Console App – even trying to add the Microsoft EntityFramework at this time produces a compiler error. However I can search for ‘Microsoft.EntityFrameworkCore’ and I get a library that seems to be fine. Trying to install the Oracle ODP Framework though gives the expected compiler issue.

At this point its probably worth abandoning this attempt.

3, Connecting to Oracle from a .NET Core App on .NET Framework

So for this attempt I’m going to try a web application because something that interests me (but I haven’t yet got my head round) is the 3 option listed for ASP.NET Core Web Application (.NET Framework).
image

I select Web Application to get something running quickly. I then added the Microsoft Entity Framework Core Library, via NuGet as mentioned in Step 2. I then also added the ODP Oracle Entity Framework (as mentioned in Step 1, this time without issue).

However clicking into Add New Item – the Data option previously used isn’t on show.

4, Connecting to Oracle from a ASP.NET MVC on .NET Framework

Switching to create a New ASP.NET MVC application on the .NET Framework and following the steps above (but using Entity Framework rather than the dotnet core version) does produce the option to have a Data object added – and following the steps detailed in Stage 1 seems to work.

The next step I guess for this would be to try the dotnet core options with a MS SQL Server database and see what the support is like. However, it looks like we’ll still be on .NET 4.6.X until some Oracle DB support comes to dotnet core.

Related Reading:

https://csharp.today/entity-framework-6-database-first-with-oracle/

https://docs.efproject.net/en/latest/platforms/aspnetcore/existing-db.html

https://community.oracle.com/thread/3903545

Java 8 and LAMBDAS

A little bit late to the party with this one. I’ve been keeping up-to-date with Java 8 (and 9) changes, but our enterprise servers tend to not to be updated early on (understandably).

However, Oracle ran a Java 8 MOOC recently (they’ve since made the videos available via YouTube). I’m particularly interested in the parallel streams as we have a long running process that may benefit from the new syntax.

An unscientific test on my own laptop comparing the standard Java loop (an example from the MOOC):

for (Student s : students) {
   if (s.getGradYear() == 2011) {
      if (s.getScore() &gt; highestScore) {
         highestScore = s.getScore();
      }
   }
}

Takes approximately 469 ms to run.

However, using the new Lambda syntax:

highestScore = students.parallelStream()
                  .filter(s -&gt; s.getGradYear() == 2011)
                  .mapToDouble(s -&gt; s.getScore())
                  .max().getAsDouble();

This comes down to 272 ms.

As I say, unscientific and a very simple use case. Be interesting to see what happens in the real world though.

Oracle sql loader investigations

Today I’ve been exposed to the world of SQL Loader. There is a process that runs on a daily basis that does the following:

1. Parse a text file and copy to a remote server
2. Use SQL Loader to append the contents of this file to a database table
3. A final process to concatenate 2 fields in the database after the insert has completed.

Step 3 is an unnecessary overhead that was introduced a while back. Steps 1 or 2 are the better areas for this fix to be implemented. Having never used SQL Loader it looked a more interesting learning experience, and heres some quick findings.

To read the value into a temporary store, use the BOUNDFILLER key word (apparently supported in Oracle 9+). However, with our implementation, then continue to process the columns in order, before finally coming back to insert the temp values into their relevant columns.

An example:

The text file looks roughly like this:

1[TAB]A[TAB]MODULE[TAB]0LTD0023[TAB]25/03/15[TAB]Username

And the database table has the following columns

ID
SEMESTER
TYPE
CODE
ACCESS_DATE
USERNAME

The SQL Loader to concatenate File Entry [0] and File Entry [3] is as follows:

(
tempID BOUNDFILLER,
SEMESTER,
TYPE,
tempCode BOUNDFILLER,
ACCESS_DATE,
USERNAME,
ID “:tempID”,
CODE “:tempCode||:tempID”
)

Found plenty of examples on the Internet, but few made reference to the fields needing to be the final ones inserted.