JMS and GlassFish 4–The Overview

As part of the latest Java EE release (version 7), the JMS component has been reviewed and revamped. I was recently fortunate enough to attend the inaugural London GlassFish Users Group earlier this week where Nigel Deakin from Oracle was presenting the latest features and developments.

This blog post is really to expand my knowledge, test some of the things mentioned, and to act as reference to my future self.

1. Getting Started

All examples in this blog post are going to be using GlassFish 4.0 and NetBeans 7.3.

1.1 Queue Creation

First of all start the GlassFish server, and then log into the Admin Console (localhost:4848). We’re then going to create our Queue and Connection Factory to handle the queue.

Next up, create the queue by navigating to Resources –> JMS Resources –> Destination Resources, and click New. Then enter the following parameters:

image

Then create the Connection Factory by going to Resources –> JMS Resources –> Destination Resources, and enter the following parameters:

image

Thats us got the infrastructure for the queue to function. (I’ve subsequently discovered there is no need for the Connection Factory, GF4 provides a default one that would be used if no others have been created!).

1.2 Sending First Message (using JMS 1.1)

Now return to NetBeans and lets create a servlet to send the first message. We’re going to use JMS 1.1 for this example, before then doing JMS 2.0 which will hopefully show some of the simplifications that have been made.

In NetBeans create a new servlet, named something like JMSOne. Then add the following code to the processRequest() method.


Connection connection = null;
ConnectionFactory connectionFactory = null;
Destination dest = null;
Context jndiContext = null;

try {
   jndiContext = new InitialContext();
   connectionFactory = (ConnectionFactory) jndiContext.lookup("jms/testQueueFactory");
   dest = (Destination) jndiContext.lookup("jms/testQueue");

   connection = connectionFactory.createConnection();

   // The 2 parameters passed here do not make any difference in GF!
   Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
   MessageProducer producer = session.createProducer(dest);
   TextMessage message = session.createTextMessage();
   message.setText("First test message!");
   producer.send(message);
} catch (NamingException | JMSException e) {
   System.err.println("Exception raised: " + e.getMessage());
   e.printStackTrace(System.err);
} finally {
   if (connection != null) {
      connection.close();
   }

   if (jndiContext != null) {
      jndiContext.close();
   }
}

Run the servlet, and perhaps have some output, either to the screen or to the GlassFish log and your message should send. (How do I know its been sent? Well, this is something that I think is lacking in GF – or I’m missing something simple. I can’t see where you can see the current status of the queue – but I’ll see if I can address this later in this blog post).

1.3 Retrieving the Message

Now we need to create a message consumer to read the message that we have just sent. NetBeans makes this incredible straight-forward to do by providing most of the code for you.

Right click the project and create New->Enterprise Java Beans –> Message Driven Beans and then enter the parameters for the queue that we created previously.

The code to retrieve the bean is entered into the onMessage() method and looks like:

@Override
public void onMessage(Message message) {
   String msg;
   try {
      msg = ((TextMessage) message).getText();
      Logger.getLogger(NewMessageBean.class.getName()).log(Level.SEVERE, msg);
   } catch (JMSException ex) {
      Logger.getLogger(NewMessageBean.class.getName()).log(Level.SEVERE, null, ex);
   }
}

1.4 Sending using JMS 2.0

So the code previous is straight-forward, but slightly long winded. Admittedly some could be removed with try with resources functionality introduced in JavaEE 7, but still. Now create a new Java Servlet, along the lines of JMSTwo.

The code to send the message now looks like this (using JMS 2.0) (note, I’m calling sendMessage() from my processRequest method):

@Resource(lookup = "jms/testQueue")
private Queue queue;
@Inject
private JMSContext jmsContext;

private void sendMessage() {
   String msg = "My JMS 2.0 Message";
   jmsContext.createProducer().send(queue, msg);
}

The code is now down to 2 lines, and 1 of those is to declare the data to send. There is no need for the session, connection factory or connection. Everything is taken care of.

2. Other JMS 2.0 Improvements

Alongside the simplifications to sending a message, JMS 2.0 has brought about a few other enhancements. These are:

  • Delayed Delivery
  • Async Sending
  • Improvements to Poison Message Handling

2.1 Delayed Delivery Example

The code for this a very straight-forward. The time is in milliseconds, so the code below delays sending the message for 60 seconds. Note this doesn’t hold the servlet from continuing. By watching my server.log for the 60 seconds after, we see the text from the message appear.

jmsContext.createProducer().setDeliveryDelay(60000).send(queue, msg);

2.2 Async Sending

This doesn’t fit well into this blog post – its for use with standalone applications outside of the Java EE Web and EJB containers. It does however allow the application to send a message and then continue processing further steps without the need to wait for a reply.

A future blog post may look into standalone applications sending messages.

2.3 Poison Message Handling

Poison Message Handling really comes into effect when a message is returned to the queue for re-processing for whatever reason. The most obvious example would be when a connection to the database cannot be obtained (e.g. due to network or infrastructure issues). An improvement that has been made in JMS 2.0 is that the number of times a message has been processed can be retrieved.

For this we’re going to update the onMessage() received bean. The code below throws a dummy exception to return the message back to the queue. After the 3rd attempt, it writes the message contents to the server.log for later processing.

By updating the receiving code to look like this, we can see the message redelivery count increasing:

String msg;
try {
   msg = ((TextMessage) message).getText();
   int deliveryCount = message.getIntProperty("JMSXDeliveryCount");
   String deliveryCountStr = "Received this message " + deliveryCount + " times.";
   Logger.getLogger(NewMessageBean.class.getName()).log(Level.SEVERE, deliveryCountStr);

   if (deliveryCount == 2) {
      Logger.getLogger(NewMessageBean.class.getName()).log(Level.SEVERE, "Message can''t be processed. Details...: {0}", msg);
   } else {
      Logger.getLogger(NewMessageBean.class.getName()).log(Level.SEVERE, "Throwing a dummy runtime exception");
      throw new RuntimeException("Dummy Exception");
   }
} catch (JMSException ex) {
   Logger.getLogger(NewMessageBean.class.getName()).log(Level.SEVERE, null, ex);
}

And the truncated version of the server.log file now looks like this (removing the stack traces for brevity):

SEVERE:   Received this message 1 times.
SEVERE:   Throwing a dummy runtime exception
WARNING:   MQJMSRA_MR2001: run:Caught Exception from onMessage():Redelivering:
SEVERE:   Received this message 2 times.
SEVERE:   Message can’t be processed. Details…: My JMS 2.0 Message

A more suitable use case for myself in this scenario is actually getting the message queue to stop processing messages in this scenario. I’d like then to build up whilst the reason for failure is detected (normally due to it being a networking / database issue).

I’ve yet to find something that I’m completely comfortable with, but provided this message doesn’t return, no further messages will be process. So prior to throwing a RuntimeException, I can put in place a Thread.sleep(milliseconds) to sleep for a period of time.

Logger.getLogger(NewMessageBean.class.getName()).log(Level.SEVERE, "Throwing a dummy runtime exception");
Thread.sleep(60000);
throw new RuntimeException("Dummy Exception");

This would allow messages to build up. The time to hold the queue for would need to be specific for the scenario being catered for. Ultimately it could be for an hour or so, and then remove the check for the number of times the message has been processed and just allow it to continue re-processing the message.

3. Monitoring the Queue(s)

3.1 imqadmin Tool

I said earlier that there is no default out-of-the-box monitoring within GlassFish for JMS. Thats not strictly true. Unfortunately (unless I’m missing something?) its not available in the Admin Console.

GlassFish does come bundled with a number of command line utilities, one of which being the Admin Manager for MQ. This utility can be found in \glassfish4\mq\bin and launched via imqadmin.

image

You then need to add a broker to localhost (Right-click on Brokers and Add…). The settings should be localhost, 7676, admin and admin (unless you’ve changed your localhost Admin password). Once added, you can then click the ‘Physical Destinations’ under your broker and the testQueue created earlier will appear.

By selecting the queue, and then clicking Properties (either via the Icon on the Toolbar or via Actions-Properties menu). This will show you details about the queue, including the number of messages sitting in it (0 at this stage).

To show some messages sitting in the queue, we can head back to our code and rename the queue being processed by the Message Bean to be testQueue2. Then running the sendMessage servlet we can populate the queue, with no listeners to pick the messages up. (For this to deploy correctly, you’ll need to create testQueue2 as per steps at the start of this post) (Further note – I also had to restart GlassFish after creating the 2nd queue – the broker and the application got completely confused and started only displaying the new queue?!? – Yet another reason to re-start GF?)

@MessageDriven(activationConfig = {
@ActivationConfigProperty(propertyName = "destinationType", propertyValue = "javax.jms.Queue"),
@ActivationConfigProperty(propertyName = "destinationLookup", propertyValue = "jms/testQueue2")
})

By running the send message servlet 5 times and then restarting the imqadmin client and connecting to localhost and viewing properties for queueTest I can see 5 message waiting in the queue.

image

3.2 Monitoring via a Servlet

This all seems quite long winded, so I put together a little servlet that would do something similar, using the QueueBrowser class. The QueueBrowser allows you to peek at the Message Queue without stripping the messages out.

I’ve yet to discover a way of listening all the queues on the server, but can’t imagine this would be difficult. Could quite easily create a JSP to ask the user to enter the queue name and then retrieve the messages and count from this servlet with some simple modifications.

try {
   QueueBrowser queueBrowser = jmsContext.createBrowser(queue);
   out.println("Queue Name: " + queueBrowser.getQueue().getQueueName() + "<br/>");
   Enumeration messages = queueBrowser.getEnumeration();
   int count = 0;
   while (messages.hasMoreElements()) {
      count++;
      TextMessage msg = (TextMessage) messages.nextElement();
      out.println("Message [" + count + "] contains " + msg.getText() + "<br/>");
   }
   out.println("Number of messages = " + count);
} catch (JMSException ex) {
   Logger.getLogger(queueViewer.class.getName()).log(Level.SEVERE, null, ex);
} finally {
   out.close();
}

4. Safety of Messages

One benefit from my experience in the MS world of MSMQ is the way messages are safely stored in cash of unexpected system restarts etc. With my 5 messages safely in the queue I’m going to restart GlassFish to see what happens to them.

And I’m pleased to say its kept them safe.

image

5. Conclusion

All in all this has been a whirlwind tour of JMS in general, but with a glance towards some of the new functionality in JMS 2.0. If anything is wrong in this post please let me know – I’m completely acknowledging my ‘learning’ position with regards to this technology.

Likewise, if anything is missing that you’d like me to research and cover, please let me know.

I’m already looking forward to the next GlassFish Users Group next month.

Evernote–My Tips and Set-up

I love Evernote. So much so some of my colleagues are wondering if I’m working on the side as a salesperson for them. Anyway, I saw a tweet in my timeline this morning asking for some tips for usage, so I thought I’d share my set-up and also some tips I gleaned along the way. I did a short post back in March about this topic, but I’ve moved on slightly since then, so this is a much fuller description and its been in place for about 4 months now.

Getting Things into Evernote

1. Your Evernote account has an email address. Its a a strange mix of characters, presumably representing an underlying unique username in their system. Too difficult to remember at any stage. So add it to your email contacts list and give it a friendlier name. Having my own domain, I’ve even set-up en@mymadeupdomain.com just so others can email me direct to Evernote when in meetings.

To get your Evernote email go to Account Settings on the Web, or in the Windows Desktop App – Tools –> Account Settings.

There is also a plug-in for Outlook so you can just click a button to add any email to Evernote.

image

Its been a while since I installed it, but a quick Google takes me to this discussion page, which should provide further help.

There are also useful annotations that you can add to the subject line to get the email into the right notebook in Evernote. By default it goes into your Inbox/default notebook. You can alter this by (using a @ to change the notebook) or add a tag (adding a #). You can also add reminders this way with an !. I very rarely use these though. For fuller, more precise details see here in this video.

2. The Android Widget. (Probably available on iOS too?). Normally when I’m walking around is when I remember things I need to do. This little widget will allow me to take notes either written, via the camera or an audio recording and send it straight to Evernote. It also records the GPS of where you were – should that be a factor.

image

(Also worth noting from here, you can add a PIN to your Evernote Mobile App to stop access in case it falls into the wrong hands. Short video here. Is this a premium only feature?)

3. Camscanner (Android App – iOS?). This is basically a scanner for your phone. It allows you to adjust what you’ve captured and then share via a range of ways, including to Evernote. As another aside, I’ve found this incredibly useful when attending conferences / lectures and taking photos of the presentation slides.

Setting Up Evernote – Notebooks and Stacks

A caveat – this is my set-up. It works for me. Its not necessarily the way. Or even the way for you.

1. Current Notebooks

My framework is based loosely on the GTD principles from David Allen. At the top of my notebook view I have 5 main notebooks – !Inbox, !IncubationIdeas, !Journal, !ToDo, !WaitingFor.

image

You’ll notice the ! marks in front. This is on purpose. It means these will stay above all my other notebooks. These are probably my most used notebooks.

  • Inbox represents things that I’ve captured but not yet processed.
  • Incubation Ideas are ideas for things I’d like to do. At some point. I may not have time now, or the time might not be quite right.
  • Journal is my attempt to capture everything I do in a day. (More on this later).
  • To Do is my main go to area. I have a separate note in here for each of the tasks that I’m aware of need action. (The title of the note is usually the next possible action and the note itself contains a checkbox (more later)).
  • Waiting For is stuff I’ve delegated but need to have an awareness of.

2. Current Project Notebooks

Sticking with the GTD principles I have a list of current projects that are happening in my life – ranging from home personal stuff to work related. Within these I tend to have at least 2 notes – ‘Project Log’ and ‘Project Actions’. Alongside these 2 notes sit anything else I’d like to refer to for future knowledge.

image

  • Project Log is something I’m experimenting with. Its essentially like a blog where I put date ordered notes about anything that happened, who was involved. It gives me a nice overview of everything that has happened in a project. (I used to keep these as separate notes, but found it a bit unwieldy).
  • Project Actions are a collect of To Do items that relate to this project. Again each action is annotated with an empty checkbox. It doesn’t always play nicely with my !ToDo notebook above, but I use the Search facility to overcome this generally (more later)

3. Other Notebook Stacks

Everything else is just for reference – stuff I may want to refer to in future or a way of storing things. This essentially takes a format of a file system – using notebooks as a folder equivalent.

Projects that are complete are moved into this area too for archiving / future referencing.

This for me is where Evernote comes into its own. The search facility is truly amazing, and was perhaps the part of Evernote that I really didn’t fully appreciate when first starting out.

Other Little Tips I’ve Gathered

1. Daily Journal. Each day I start a new note. Its created via a IFTTT recipe I’ve got in place so its automatically created and ready to use. The basic format is as below:

image

Before really starting work each day I note down the 3 overriding actions I’d like to get completed today – this comes from the principles outlined in Getting Results the Agile Way. I also note down any meetings, bullet point any actions I’ve performed during the day and I attempt to keep a food diary.

2. Search for Empty Checkboxes. Anything that requires action by me (either in my ToDo notebook or under one of my project notebooks) has a checkbox put against it. Once its complete I check the checkbox. This allows me to search for outstanding actions. To create this search (in the latest Windows Desktop App) click File->New Saved Search. Give it a title and set the criteria to:

todo:false

3. Tags. I have used. And then I’ve removed. And then I’ve added back. Currently I’m not using them. I can see their benefit but it just adds an extra layer of complexity to my set-up.

4. Subscribe to your blog via Evernote Email. Have your own blog? Then subscribe to it using your Evernote email. That way every time a blog post is made, you’ll have an email back-up copy in your Evernote without having to do anything.

Performance Testing with JMeter 2.9 Book Review

Disclaimer: I was provided with a copy of the book free of charge in exchange for a review. The views expressed below are mine.

JMeter

I’ve blogged previously about my love for the simplicity of JMeter, but also acknowledged my lack of in-depth know-how and its something I should really get more to grips with. I hoped that this book would take me to the next level with JMeter, and indeed the book arrived at a very timely stage for a project I’m currently working on.

The book begins with a very good introduction into why you’d want to use JMeter (or similar tools) in the first place, and presents a scenario where the benefits of it use can be seen. It then proceeds to guide through the installation of JMeter. All good so far. Unfortunately it then starts to go into what I refer to some more advanced topics before starting to use the application for the first time. These topics deserve a mention, but perhaps appear at the wrong place in the book. However glossing over them and making a mental note for where they exist for future reference seems to suffix.

The book however did present a few nuggets of information for myself, broadly split into the following categories:

  • How to record a test plan to simulate proper use of a website
  • The session / authorisation components
  • The distributed testing model to overcome local machine performance issues
  • The ability to directly test a database connection

And for these reasons alone has proved a very worthwhile read for myself. The book also goes into detail on a number of other topics / areas, many unfortunately not relevant to me at this moment, but were interesting none the less, and just re-confirmed my initial impressions that JMeter is very powerful tool with flexibility to accommodate many situations. I’ve made a mental note of the topics covered and will no doubt refer to this book to exploit them should the need arise on future projects.

Whilst the book is very good overall, there are a couple of weaknesses that need to be highlighted. I’ve bought a number of titles in the past from PacktPub, but they’ve lacked the final polish that other publishers appear to have. This also appears apparent in this book – indeed the first step-by-step exercise that appears could be set out so much clearer with just a little thought into the layout and numbering of the steps. At one point in another exercise I found myself referring to videos on YouTube for clarity in order to overcome the problem I was facing. However, whilst this could be classed as a weakness, some (including myself) may see this as an advantage – if everything was plain sailing would we really be engaging our minds and learning?

So all-in-all, I thought a very good book. I’ve certainly learnt a lot about a very useful tool, and its encouraged me to make more use of it on current and future projects. I now feel in a position to exploit its usefulness ever further than the simple tests I had been running previously. Indeed this week I think I’ve used JMeter more times and to better effect then I have since my original JMeter blog post back in 2010.

Going Paperless

Last year I thought about having a new years resolution about going paperless. The length of time to write that previous sentence could probably be longer than the effort I dedicated to that task last year.

But this year I’m going to give it a shot. Not really as a resolution, but more because I used Evernote to handle my ToDo List, attempted a new way of doing things (FWIW I used my own take on GTD, I then tried to be more expressive with TSW which didn’t really work for me), which left everything in a bit of chaos. Whilst tidying Evernote out, I decided to Google around and re-read some stuff about going paperless.

So as part of my tidy up of my To Do List, I’ve decided to look at how I use Evernote and attempt to basically put everything in there from now on.

Setting Up Evernote

1. !Inbox default notebook. Everything will go here. Any email requiring more than 5 minutes work will end up for re-evaluation. Any photos I take from my phone. Any notes I take on my phone.

2. !Journal – This is my day-to-day recording of what happens each day. I start with the 3 things I want to accomplish that day (hat tip to: Getting Results the Agile Way). I also do my best to summarise what happened that day, and record my daily exercise routine.

3. Then 3 Notebook Stacks:

  • Personal Life (contains any personal documentation, car stuff, exercise articles etc…)
  • Masters Degree (with a seperate notebook for each module and my disertatation)
  • Work (seperate notebook for each project, conferences and a general folder)

4. Finally a ToDo notebook, with an individual note for the next action. I also use tags here to essentially break tasks into those relating to Home, Work or my Masters.

Getting ‘Stuff’ In

Emails

As per the GTD guide, any email to any of my accounts gets the following treatment:

  • Resolve in less than 5 minutes, now – Do It Now (and then decide whether to delete or file. If I may require in future, I email it Evernote).
  • Defer It – Send straight on to some else.
  • Don’t Do It – Delete it.
  • Do It Later – Send it to Evernote !Inbox notebook to classify and sort out.

There are 2 ways I do this. Evernote has an excellent MS Outlook Plug-In – so any work emails get a click of the plug-in and they get transferred to Evernote. Other accounts are web-based, and I simply forward them to my Evernote address (Note: create a contact called Evernote). I am considering on a couple of my less frequently used emails to try and do auto-forwarding.

Paper

Any paper I receive through the day (and to be fair, its not much), a similar process exists as email. Bin it (well, recycle it), do it or send to Evernote Inbox for later. In order to help with this, I’ve got 2 applications on my Android phone. CamScanner works really well at capturing high quality, readable images – so anything with important steps I capture via that route. It integrates nicely with Evernote.

If the quality isn’t required, then my Evernote app comes with a widget to quickly capture an image and send to Inbox.

Ideas

The Evernote widget also has a quick link to create a text note or an audio note. These notes also captured the GPS location, so (should I need to?) I’d be able to figure out where I was when capturing the note.

All That Old Paper

Well, nope. I haven’t got time really to go through it all. I also don’t own a scanner (yet!). My thoughts at this point is to capture stuff as it arrives, and then if I go hunting for anything, I’ll capture the located document at that point.

Thats It

So here we are. I’m not really sure where this will lead, but I’ll give it a shot. Probably best to review again in 6 months time for how things are going. No doubt they’ll be plenty of lessons along the way.

Soft Systems Methodology–My Understanding?

Disclaimer – I’m learning about this subject area on a module this semester so my understanding, and example given below, my be incorrect. I doing this post to gain feedback from the wider community, my tutor and fellow students.

Underlying Principle

Identify a problem. Then step back from that problem and look at the overall structure. Ask simple questions like why are doing this? What are we trying to achieve? How *should* we be doing this?

A Simple Example

So in my world of trying to make everything complex simple, leads me to bringing this subject area back to the area of football. I won’t be going in-depth so a understanding of the sport won’t be required!

Step 1 – Identify the problem (ignore the solution)

The Problem: We’re not winning football matches / trophies. Its the managers fault – sack the manager!

Step 2 – Rich Picture

Draw a rich picture showing the entire solution area – to understand connections between various stakeholders and the wider environment.

Step 3 – Step Away from the Problem:

What Are We Trying to Do: Win trophies. Be successful.

How could we do this: Ideally, we’d have the best goalkeeper, best defenders, best midfielders and best forwards in the league, if not the world. We’d also have excellent (although not necessarily the best) squad to cover any injuries and suspensions. We’d also have a very good manager to get best out of our players, and apply tactics to limit the opposition strengths and exploit their weaknesses.

Step 4 – Identify the ideal model/system to achieve this

Step Description
1 Best goalkeeper
2 Best defenders
3 Best midfielders
4 Best attackers
5 Very good squad to cover suspensions and injuries
6 Good manager to make best of our strengths and exploit other teams weaknesses

Alongside the ideal model (conceptual model) we need to consider constraints. Examples in this scenario could be money, expectations of success sooner rather than later.

We would also need to consider how to monitor progress – we could use statistics like the league table compared to this point last season, individual player stats, perhaps fitness levels.

Step 5 – Gap Analysis (Compare the Ideal model to What is Currently in Place)

Step Description Currently Have? Do Well? Further discussion
1 Best goalkeeper Yes -
2 Best defenders Yes -
3 Best midfielders Maybe Still best, but some getting old.
4 Best strikers No Discuss further
5 Best squad No Discuss further
6 Best Manager Yes -

Step 6 – Hold Meeting to Have Discussion Points

Here we can see these likely revolve around Steps 3,4 & 5 in the model. Discussion around constraints will happen here – an immediate example would to buy 2 new forwards, but that could cost £100m – is the best investment of money, and is the money available.

#lak13 (Late) Assignment 1–Logic and Structure Assignment

Ok, due by the 4th March, so I’ve missed the deadline by some distance. But hey, its not graded and I quite liked the idea behind it. With a couple of deadlines passed on my actual University course I may have more time to dedicate to this MOOC.

1. What do you want to do / understand better / solve?

I’ve got some grand ideas. But I’ve decided to start with something relatively simple to at least get started. In essence I’m looking to identify the ‘At Risk’ student – ultimately so support mechanism can ‘kick in’ and help reduce the odds of the student dropping out.

2. People involved? Social Implications?

It would be great to have a team of various people from around the University to help develop the idea, but I’m not sure quite how I’d go about that. They’d have a much greater input into any missing elements / systems.

The only other major implication is the identification of the right person / set of people who could start the wheels for the support mechanism. I would imagine this would be a faculty / school admin person in the first instance, before probably being the course / module administrator for identified students.

3. Brainstorming – How could this be solved?

My initial trail of thoughts have been around identifying those students we haven’t ‘seen’ for a period of time (6 weeks by default – coincides nicely with the HESA return process). But ‘seen’, I mean haven’t accessed various systems, starting with:

  • Have swiped the attendance system, but not within the last 6 weeks.
  • Not logged into the VLE and / or accessed any teaching resources in the last 6 weeks.
  • Not submitted to an assignment in the last 6 weeks

Other mechanisms around library book usage could be identified.

4. Potential data sources – any issues? How ‘clean’ is the data?

  • Would need the list of ‘current’ students. Held centrally, a number of primary keys including the Student Number.
  • Attendance Monitoring System – Primary ID is the Student Number
  • VLE Monitoring Logs – Primary ID is the Username. But a username to SRN resolver in is place.
  • Assignments System – Number of primary keys, including Username and Student Number

#lak13–Getting Data In (and Getting Data Back Out!)

So the previous blog post talked more about the high-level view of how data transfer could be performed within an institution. Now we’ve (potentially) sussed out how systems can link, we can turn our attention to the actual movement of data.

When integrating systems, the top level action in any meeting is getting the blasted data into the system. After all, I guess that is the goal of any integration project. However, I prefer to take a backwards step and look at how to get the data out. This is 3-fold really:

  1. Its nearly always easier to read data than write. (Prove the simplest thing works)
  2. Having the ability to read allows for automated testing of writing
  3. It allows for long-term monitoring and much more functionality to enhance the student experience (but more this later)

Using Web Services

So in front of every system I’ve integrated I attempt to get the standard CRUD services in place. CRUD being Create, Read, Update, Delete. (After the request for Read to be done first, people normally start thinking I’m real idiot asking for Delete especially if data should never be deleted. But my thinking is you’re in there learning now, put the delete in place whilst testing, put it under source control and then remove the functionality – rather than needing to come back to it in 5 years time when everything is forgotten).

Read Providing Mechanism for Analytics

So whilst Create and Update get the data in, from an analytics point of view it is the read that bears the most interest. So using the library system as an example, we could probably place a read service that takes the Unique User ID and returns the users current status, along with their history.

From an analytics point of view, we could look at a number of students, comparing and contrasting grades and looking at the number of books they’d borrowed from the library (and/or how they much they owed in late fines). Did all the students who did very well on a single module all borrow the same book? If so, does the library have enough copies of this book for everyone on the course? (Of course, it gets difficult to determine if others had bought the book rather than borrowed it, or used an eBook instead). But there could be patterns here.

Could we also detect the student who hasn’t used the library? Are they at risk of dropping out? Is there a training need about how to locate books and check them out? Of course, like above, they may have bought there own copies or alternatively just used reference only materials in the library. But then we could use the swipe system to determine if they’ve ever swiped in whilst in the library / LRC.

Achievements System

This read structure could of course fit nicely with a rewards / achievements structure. A number of first achievements could be devised by the University, such as the first time you’ve taken out a book – a badge appears on your VLE page (I’m not a 4Square user, but I think they use this concept for different types of check-ins). This could of course further add to the analytics questions above – if you’ve not borrowed a book but everyone on your module / course has – are you at risk of dropping out? Can we intervene? Not only displaying you have been awarded this badge – we could display something like ‘a lot of your classmates have been awarded the ‘First Book Check-Out Badge’ – want to find out how to win yours? Click here – linking off to video and resources explaining how to locate books, check them out and where to go for further support.

Enhancing Student Experience

Provided the achievements are well thought through – they really could contribute to the student experience. Firstly some services that are offered by the University are not always known to students. This achievements page could highlight other awards to research – showing the range of services available, along with links for how to use them (and ultimately award an achievement). (It would also serve a reference page to help when they forgot the procedure for booking a Group Study Room for example from one year to the next).

Finally though, the read services in front of these satellite systems would also contribute to the availability of a student portal – letting students know what they need to know, now. When a student logs into the VLE, a read request could be made to the library system. They could be notified of the amount of money they owe (with a link to pay), and when their next book is due back – the sooner it is, the more prominent this message becomes.

And boringly, they aid support of systems

I should put this in small print. But by having read services available, tools to help diagnose student status’ across various systems can be developed to help resolve issues – find out where faults have occurred. A diagnostic tool if you will. If your Helpdesk could see your status across all systems with just entering your ID number / username – how much easier would their life be?

Oh, the read service would indicate if the system was up or down – help report to users that the system is unavailable, and allowing support staff to proactive to faults, rather than reactive.

The next blog post might actually get round to crunching some data, and see some really stuff happen – but that’ll need to keep until tomorrow.